Storage options for bandersnatch¶
Bandersnatch was originally developed for POSIX file system. Bandersnatch now supports:
POSIX / Windows filesystem (transparently via pathlib)
Filesystem Support¶
This is the default mode for bandersnatch.
Config Example¶
[mirror]
directory = /data/pypi/mirror
storage-backend = filesystem
# Optional index hashing to store simple HTML in directories
# Recommended as PyPI has a lot of packages these days
hash-index = true
Amazon S3¶
To enable S3 support the optional s3
install must be done:
pip install bandersnatch[s3]
Add a
[s3]
section in the bandersnatch config file
You will need an AWS account and an S3 bucket
Config Example¶
[mirror]
# Place your s3 path here - e.g. /{bucket name}/{prefix}
directory = /my-s3-bucket/prefix
# Set storage-backend to s3
storage-backend = s3
# Provide s3 style path - e.g. /{bucket name}/{prefix}/{key}
diff-file = /your-s3-bucket/bucket-key
[s3]
# Optional Region name - can be empty if IAM are set
region_name = us-east-1
aws_access_key_id = your s3 access key
aws_secret_access_key = your s3 secret access key
# Use endpoint_url to indicate custom s3 endpoint e.g. like minio etc.
endpoint_url = endpoint url
# Optional manual signature version for compatibility
signature_version = s3v4
Serving your Mirror¶
S3 Bandersnatch mirrors are designed to be served with s3 static sites and can also be used with the Amazon CDN service or another CDN service.
I assume you have already set up an AWS account and S3 bucket, and the Bandersnatch sync job has successfully ran.
Enabling website hosting for the bucket¶
When you enable the website hosting for a bucket, this bucket can be viewed as static website. Using the s3 domain or your customized domain.
Please read Amazon documents to get detailed instructions
Most cloud provider who provide a s3-compatible service will provide this service as well. Please consult to your service assistant to get detailed instructions.
Use CloudFront or other cdn service to speed up the static mirror(optional)¶
If your mirror is targeted to global clients, you can use CloudFront or other CDN service to speed up the mirror.
Please read Amazon documents to get detailed instructions
Set redirect or url rewrite in CloudFront or other cdn(optional)¶
In most cases, packages and index pages are all inside /my-s3-bucket/prefix/web
, if you set up a steps above, you should be able to use the mirror like this:
pip install -i my-s3-bucket.cloudfront.net/prefix/web/simple install django
But there are two main disadvantages:
The url is quite long and exposing the structure of bucket.
Users will be able to view all content in the bucket, including bandersnatch todo file and status file.
It is strongly recommended to set redirect or url rewrite for CDN. Please contact your service assistant for detailed instructions.