idp_proxy: Author: Seven Du from [Idapted Inc.](http://www.idapted.com) typical usage is to work with nginx cache protected files. Your background app respond to check if a user has the proper rights/permissions to access some certain files, and return a X-Accel-Redirect header tell nginx to serve the file. If a file is found in the local storage, then nginx serve it directly, or fetch from a Amazon S3 bucket and cached locally. Then you may want another cron like job to delete old files in the local storage. 1) turn on sendfile on nginx conf: sendfile on; 2) make you app send the sendfile header: php: header("X-Accel-Redirect: /file-internal/filename.jpg"); Ruby/Rails: head(:x_accel_redirect => "/file-internal/filename.jpg", :content_disposition => "#{disposition}; filename=\"real_filename.jpg\"") 3) config Nginx for internal access only: location /file-internal { internal; alias /tmp/idp_proxy; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Uri $uri; # you don't need to turn on proxy store since idp_proxy will do this for you # proxy_store on; # proxy_store_access user:rw group:rw all:r; # proxy_temp_path /tmp/nginx_temp; proxy_set_header X-debug1 $request_filename; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; if (!-f $request_filename) { proxy_pass http://127.0.0.1:8910; } } 4) the code depends on [mochiweb](http://code.google.com/p/mochiweb/), and [ibrowse](http://www.nobugs.org/developer/s3erl/). So get them and either change the symbol links in deps dir or delete the symbol links and put them in place. 5) idp_proxy use a modified version of [s3erl](http://www.nobugs.org/developer/s3erl/) to fetch files from s3. Changed to use [ibrowse](http://www.nobugs.org/developer/s3erl/) instead of inets:http to stream file directly to disk. ibrowse options is configurable in deps/ibrowse/priv/ibrowse.conf 6) configure and compile idp_proxy cd idp_proxy cp src/idp_proxy_sample.hrl src/idp_proxy.hrl edit src/idp_proxy.hrl INTERNAL_PATH: relative path will be removed from the url and concatenate with DOC_ROOT to get the local path TMP_PATH: temp path to store files downloading from S3 DOC_ROOT: should meet the alias in the nginx conf, or can be anywhere if you turn on proxy_store 7) run make ./start-dev.sh 8) now open another terminal window and run test/test.sh TODO: * serve immediately to nginx in chunk without waiting for the complete of s3 download