WangZhenfei / file-transfer-accelerators

Discussions

• http://www.quora.com/What-kind-of-technology-does-Filecatalyst-and-Aspera-uses-to-fast-file-transfers
• http://serverfault.com/questions/300086/file-transfer-alternatives
• http://serverfault.com/questions/111813/what-is-the-best-way-to-transfer-a-single-large-file-over-a-high-speed-high-lat
• http://forums.smallnetbuilder.com/showthread.php?t=10950
• http://www.networkworld.com/news/2012/111312-file-transfer-tools-264194.html
• http://www.quora.com/James-Lewis-10/answers
• http://www.linkedin.com/groupItem?view=&gid=4341810&type=member&item=99631847&commentID=76026411&qid=51d0f011-7deb-4b74-b7c9-a8e5abdae9b7&trk=groups_most_popular-0-b-cmn&goback=%2Egmp_4341810#commentID_76026411

File Catalyst:

• http://www.filecatalyst.com/what-problems-are-we-solving
• http://www.filecatalyst.com/collateral/Accelerating_File_Transfers.pdf
• Customers:
  • http://www.filecatalyst.com/quicklink-and-filecatalyst-together-on-ios-and-iphone
  • http://www.filecatalyst.com/sending-large-media-files-how-free-services-stack-up (February 14, 2013)
  • http://svgeurope.org/blog/headlines/shapeshifter-taps-filecatalyst-for-large-media-file-transfer/
• Critic:
  • http://etherealmind.com/file-catalyst-unlimitech-software-dishonest-comments/
• Pricing:
  • http://www.filecatalyst.com/products/filecatalyst-workflow/workflow-pricing
• Videos

JScape

• http://www.jscape.com/Portals/26878/whitepapers/how_to_boost_file_transfer_speeds_with_aftp.pdf

• http://www.jscape.com/blog/bid/76604/Accelerated-File-Transfer-In-Action

• http://www.jscape.com/products/accelerated-file-transfer/

• http://www.jscape.com/blog/bid/99124/4-Popular-Large-File-Transfer-Methods-With-Big-Deficiencies

• http://www.jscape.com/blog/bid/94894/Solving-the-Problem-of-Big-Data-Transfers-in-the-Film-Industry

• http://www.jscape.com/products/file-transfer-servers/jscape-mft-server/

• http://www.jscape.com/download-jscape-mft-server/

• http://www.jscape.com/blog/?Tag=Case+Studies

• Version History

• Managed File Transfer and Network Solutions

• Download:

  • http://www.jscape.com/downloads/anyclient
  • http://www.jscape.com/downloads/jscape-mft-server/
  • http://www.jscape.com/downloads/file-transfer-command-line

• Docs: http://www.jscape.com/support/documentation/

  • http://files.jscape.com/ftcl/docs/html/
  • http://files.jscape.com/secureftpserver/docs/index.html

• Pricing:

  • http://store.jscape.com/index.php?main_page=index&cPath=1_2

DataExp:

• http://www.dataexpedition.com/expedat/?gclid=CLqAnZKatbkCFYKS3godiEkAXw

Aspera + Amazon

• http://readwrite.com/2011/09/27/aspera-sends-your-data-directl

Comparison:

• http://www.thinkmind.org/index.php?view=article&articleid=icns_2013_2_40_10167
• http://www.thinkmind.org/download.php?articleid=icns_2013_2_40_10167
• http://www.filecatalyst.com/pride-in-our-work
  • TIXStream
  • FC Direct
  • ExpeDat

Companies:

• http://www.tixeltec.com/index_en.html
• http://www.tixeltec.com/ps_tixway_en.html
• http://agilesys.com/bridgeworks/

S3 Fuse

• https://code.google.com/p/s3fs/wiki/FuseOverAmazon
• https://code.google.com/p/s3fs/issues/detail?id=24
• http://www.idevelopment.info/data/AWS/AWS_Tips/AWS_Management/AWS_20.shtml#s3fs Limitations
• http://juliensimon.blogspot.de/2013/08/howto-aws-mount-s3-buckets-from-linux.html

s3ql

• https://code.google.com/p/s3ql/

Problems/Solutions with AWS + S3

• http://www.jscape.com/blog/bid/94768/A-Faster-Way-to-Send-Big-Data-to-the-Cloud
• http://aws.typepad.com/aws/2010/12/amazon-s3-object-size-limit.html
• http://www.filecatalyst.com/what-to-know-before-moving-your-data-to-the-cloud
• http://www.turnkeylinux.org/blog/exploring-s3-based-filesystems-s3fs-and-s3backer
• http://www.rightscale.com/blog/cloud-industry-insights/network-performance-within-amazon-ec2-and-amazon-s3

Airship:

• http://help.useairship.com/wiki/Home
• http://help.useairship.com/wiki/Installation
• http://help.useairship.com/wiki/License
• http://help.useairship.com/wiki/FAQ
• http://www.useairware.com/#!about/c1nmv
• http://www.useairware.com/#!client-stories/cmv5
• http://www.useairware.com/#!clients/c1szy
• http://help.useairship.com/images/4/44/AIRShip_technical_guide_v_1_2_2-2.pdf
• http://forums.smallnetbuilder.com/showpost.php?s=a429954b3fac433340e1f57b6ef3c8d2&p=74111&postcount=4

Open Source:

• How to transfer large amounts of data via network.
• http://www.filecatalyst.com/open-source-fast-file-transfers
• http://coolpandaca.wordpress.com/2012/10/30/hscp-udt-udp-a-better-way-of-scp-and-file-transfer-over-the-network/

mrsync (multicast remote sync)

• http://mrsync.sourceforge.net/
• http://sourceforge.net/projects/mrsync/files/mrsync/

RBUDP

• http://www.evl.uic.edu/cavern/RBUDP/Reliable%20Blast%20UDP.html

GridFTP

• http://en.wikipedia.org/wiki/GridFTP
• http://www.globus.org/toolkit/docs/latest-stable/gridftp/
• http://www.globus.org/toolkit/docs/latest-stable/gridftp/gridftp.pdf

bbcp

• http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html#bbcp

UDT: Breaking the Data Transfer Bottleneck

• http://udt.sourceforge.net/
• http://udt.sourceforge.net/doc.html
• http://sourceforge.net/p/udt/git/ci/master/tree/
• http://blog.verycloud.com/blog/
• http://www.hpcwire.com/hpcwire/2009-12-08/open_cloud_testbed_wins_bandwidth_challenge_at_sc09.html

Sector/Sphere

• http://sector.sourceforge.net/
• http://sector.sourceforge.net/pub/Sector-cloudcom-tutorial.pdf

Tsunami-UDP

• https://github.com/rriley/tsunami-udp
• Using Tsunami UDP in AWS- How it helps, Configuration and Comparison
• http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113.4551&rep=rep1&type=pdf
• http://www.csm.ornl.gov/~dunigan/netperf/udp/

FTP Streaming To S3

• node-formidable/formidable#227 Is it possible to stream directly to S3 without uploading to uploadDir first?

  • No. The S3 API requires you to provide the size of new files when creating them. This information is not available for multipart/form-data files until they have been fully received. This means streaming is impossible.

  • No. The S3 API requires you to provide the size of new files when creating them. This information is not available for multipart/form-data files until they have been fully received. This means streaming is impossible.

• https://github.com/felixge/node-formidable

Programming with S3

• http://stackoverflow.com/questions/6774201/how-to-upload-a-file-from-the-browser-to-amazon-s3-with-node-js-express-and-kn
• https://github.com/LearnBoost/knox
• https://github.com/nathanoehlman/knox-mpu
• https://github.com/goodeggs/knox-copy
• https://github.com/segmentio/s3-lister
• https://github.com/nathanpeck/s3-upload-stream

Aspera on Amazon S3

• http://cloud.asperasoft.com/aspera-on-demand/aspera-on-demand-for-amazon-web-services/

GOLANG

Stream directly to S3

• http://godoc.org/github.com/kr/s3/s3util

• http://godoc.org/github.com/kr/s3/s3cp

• https://github.com/brettweavnet/gosync

GoAMZ

• http://bazaar.launchpad.net/~goamz/goamz/trunk/files/head:/s3/