Giters

snarfed / bridgy-fed

🌉 A bridge between decentralized social network protocols

Home Page:https://fed.brid.gy/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

support Bluesky / AT Protocol

snarfed opened this issue · comments

commented

Background: https://bsky.social/ , https://atproto.com/docs, https://github.com/bluesky-social/atproto.

...switched to the now label for tracking roadmap and current tasks.

Blocking:

Nice to have:

Manual tests:

Implementation tasks:

  • Fully implement ATProto protocol class
  • #630
  • map object ids to originals with Object.copies. (in Object.as1? or Protocol._targets? or...?)
  • add User.copies, use it similarly to map actor/author etc targets
  • receive task queue. (later hopefully we'll migrate incoming webmentions and AP inbox deliveries to this too, but that's complicated.)
  • poll for notifications periodically, per user (bluesky-social/atproto#1538 (reply in thread) )
  • #694
  • convert ids and handles, both directions
  • serve WebFinger for ATP users
    • milestone: BF user profiles are visible in the fediverse
  • #647 (bluesky-social/atproto#1650)
  • create _atproto TXT DNS records on the fly in GCP DNS (000b08f)
  • send requestCrawl to the BGS on startup
  • ATProto.send: ignore Accepts of Follows
  • milestone: successfully federate a non-Bluesky user into the federation sandbox
  • milestone: successfully federate a Bluesky user to another protocol
  • milestone: successfully federate a non-Bluesky post into the federation sandbox
  • milestone: successfully federate a federation sandbox post to another protocol
  • decide whether we have a BF protocol per lexicon/app, eg Bluesky, or just one for all of ATProto. We'd prefer the latter, but we need some app-specific semantics. Not sure yet if we can avoid all of those...
  • snarfed/lexrpc#8
  • [ ] backfill k256_pem into existing Web and ActivityPub users
  • XRPC: https://lexrpc.readthedocs.io/, https://github.com/snarfed/lexrpc
    • [ ] Validate data against lexicon schemas (optional)
  • Data conversion to/from app.bsky lexicons in granary:
  • Bridgy Fed serves most app.bsky read XRPCs, eg https://fed.brid.gy/xrpc/app.bsky.feed.getAuthorFeed?author=snarfed.org
  • finish unifying protocol.receive across protocols?
  • Federation
    • MST
    • signed commit chain
    • blob serving
    • guarantee that commit chains we serve are immutable (except for rebases)
    • ingest/sync other PDSes' repos
    • [ ] DID resolution
    • [ ] did:web serving?
    • com.atproto.sync XRPC methods:
      • [ ] getBlob
      • [ ] getBlocks
      • getCheckout
      • [ ] getCommitPath
      • getHead
      • getRecord
      • getRepo
      • [ ] listBlobs
      • listRepos
      • [ ] notifyOfUpdate
      • [ ] requestCrawl
      • [ ] subscribeRepos
    • com.atproto.server XRPC methods:
      • [ ] createAccount
      • createSession
      • getSession
      • refreshSession
    • com.atproto.repo XRPC methods:
      • applyWrites
      • createRecord
      • deleteRecord
      • describeRepo
      • getRecord
      • listRecords
      • putRecord
      • [ ] rebaseRepo
      • [ ] uploadBlob
  • UI
    • support multiple user (protocol) types
    • support Followers with different protocols
commented

Looks like https://bsky.social/ may be the backend serving the app.bsky XRPC API for the iOS app. Most of the XRPC methods require auth though, eg https://bsky.social/xrpc/app.bsky.actor.getProfile?user=snarfed@bsky.social.app returns 401 Authentication Required, and they haven't documented auth yet. Could probably dumpster dive for it in the repo if I get bored enough!

commented

Movement! They've been working out and implementing federation, eg in https://github.com/bluesky-social/indigo , and they're about to invite more people to the beta app.

commented

They still haven't published yet, but I'm digging into this based on com.atproto.sync.getRepo/updateRepo. Eg {services.atproto_pds.endpoint}/xrpc/com.atproto.sync.getRepo?did={did}&earliest={cid_last_retrieved_root}, and from their chat:

Assuming you're on the one.social PDS and your friend's is on two.social using the handle friend.two.social.
When you tell your PDS to follow your friend, the PDS will call the com.atproto.handle method on two.social with your friend's handle, will get their DID and the CID of their repo and can then call com.atproto.sync.{get,update}Repo to get their posts etc.
When your client then calls your PDS to show posts etc, your friend's posts should be included with the rest.

commented

First pass at MST implementation is done! Now working on implementing the com.atproto.sync XRPC methods..

commented

(PDS implementation is in https://github.com/snarfed/arroba/ )

commented

Blue Sky, the same service that is lead by the guy who let the government access Twitter when he was in charge? I HIGHLY OPPOSE the integration of Blue Sky service because of this. Perhaps it's time to create a rolling fork that purposely removes Blue Sky and potentially other networks from the plugin.

Yeah, I'm unsubscribing from this product altogether. I won't support this in any way shape or form.

commented

Sorry to hear that! You're obviously welcome to fork or stop using Bridgy Fed at any time, that's your prerogative, and the beauty of both open source code and free (ie no cost to you) services like this.

Fwiw, Jack and others at Twitter initially kicked off the Bluesky project, but neither he nor Twitter have ever led it or had meaningful ownership or individual control over it since then. More: https://snarfed.org/bluesky-corporate-ownership-and-structure

commented

Moved to snarfed/arroba#2, and the BF-specific bits to #512.

commented

Is the intention just to allow following websites from Bluesky (like already live for ActivityPub), or to allow following Bluesky users from ActivityPub? If the latter, how will opt-in be handled?

I would personally find the ability to follow Bluesky users from Mastodon very useful, but I can see it being controversial for a number of reasons.

commented

Thanks for asking! The intent is to expand Bridgy Fed to be able to bridge any decentralized social protocol to any other protocol, eg web sites <=> Nostr, ActivityPub <=> Bluesky, and all other combinations. You can see some of the detailed background prep work in https://fed.brid.gy/docs#translate, along with tracking issues like #512 and #529.

Glad to hear you'd find it useful! And yup, I definitely know that it may be controversial. That's definitely not my intent, and I plan to be as careful as possible, and take all feedback into account, including experiences with existing bridges like https://mostr.pub/ ...but I expect I may still take some heat. I guess I'm ok with that.

commented

Reopening for tracking, since we haven't actually shipped this yet. 😆

commented

Super cool! Just to clarify, is this specific issue meant to introduce bridging between ActivityPub and Bluesky?

commented

@ghobs91 yes! Contingent on Bluesky turning on federation in prod, of course.

Latest status: https://snarfed.org/2023-07-11_bridgy-fed-status-update-2

commented

Oh and notably this would bridge Bluesky to all of the protocols that BF supports. Right now that's just AP and web (microformats2 + webmention), but we hope to add others too, eg Nostr.

commented

Big milestone yesterday, https://github.com/snarfed/arroba is now successfully federating with the ATProto sandbox! Demo PDS is https://arroba-pds.appspot.com/ , user is arroba5.snarfed.org. Woot!

commented

Thinking through architecture here. With the other protocols we're working with so far, individual objects are at most loosely connected. We can federate profiles, follows, posts, etc independently of each other.

ATProto, on the other hand, ties all of a given user's objects together more tightly in their repo. When a non-ATProto user follows or otherwise interacts with a non-ATProto user, we can construct their repo on demand, but that's heavyweight. Should we construct them for everyone, ahead of time?

...except most of the time we'll be seeing a given non-ATProto user for the first time, so we need to be able to construct repos on the fly anyway. So maybe we just start with that, and see whether/how we can optimize later.

commented

Big milestone, shipped Bluesky => fediverse user discovery! Search for @[handle]@atproto.brid.gy (eg @jay.bsky.team@atproto.brid.gy) in any fediverse server.

commented

Big milestone, shipped Bluesky => fediverse user discovery! Search for @[handle]@atproto.brid.gy (eg @jay.bsky.team@atproto.brid.gy) in any fediverse server.

Amazing! I only seem to be able to find @jay.bsky.team@atproto.brid.gy though, if i try any other bluesky handle it doesn't work.

commented

Same, WebFinger gives this response for other users:

<!doctype html>
<html lang=en>
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>None is not a valid atproto id</p>
commented

Argh, looks like I deployed a bug that affects *.bsky.social handles specifically. Thanks for the nudge, fixing now.

commented

Fixed!

commented

Yup works now! Great work my friend

commented

Thank you @ghobs91!

After that MR I have confirmed that Bluesky to ActivityPub to Nostr integration does in fact work! 😂

https://coracle.social/nprofile1qqsp2cqnc62az3tjf9lntzr7msz43m4clxmvtv5yw8lk0m3q5zcpjvqpz4mhxue69uhhyetvv9ujumt0wd68ytnsw43qc6mzkz

commented

I get an Upstream server request failed: 500 Server Error: Internal Server Error for url: https://atproto.brid.gy/xrpc/com.atproto.repo.getRecord?repo=did%3Aplc%3Asvu7ro7knzh3eovtpst2socq&collection=app.bsky.actor.profile&rkey=self when trying to query a user on a third-party PDS (like mine at @vriska.pds.vriska.dev). Is this expected?

commented

@leo60228 hah, Bluesky support definitely isn't stable or officially supported yet. I'm curious how you got to that error though. What specifically did you do, and where?

commented

Just by querying the ActivityPub URL with WebFinger and using curl.

commented

Thanks! We should be using your PDS's hostname from your DID doc, https://plc.bsky-sandbox.dev/did:plc:svu7ro7knzh3eovtpst2socq , but evidently we're not. I'll file another issue.

commented

Milestone: got follows working!

3-follow 4-new-follower-in-mastodon
commented

Woo, this is done! All basic interactions - user discovery, following, posts, replies, likes, reposts - are now working, both directions. Not real Bluesky, only in the federation sandbox; real Bluesky will have to wait until they turn on full federation. Still though, exciting!

commented

Previous progress reports: https://snarfed.org/2023-11-07_bridgy-fed-status-update-8, and the "previously" links at the bottom there.

commented

I guess I should leave this open until we can actually ship it 😎

commented

Is this functionality only enabled for some accounts? My Mastodon and ATProto sandbox accounts can see each other, but follows and posts don't seem to be going through in either direction.

commented

@leo60228 it's not on for any accounts, hence "until we can actually ship it" and "Not real Bluesky, only in the federation sandbox" 😁.

commented

I meant in the sandbox, I know that real Bluesky accounts won't work yet.

commented

Ah! No, it's not fully on in the sandbox either. I don't currently have plans to turn it back on there and leave it on, since the sandbox is just for testing and gets wiped semi-regularly. Glad you're interested though!

commented

Is there anything I can do to contribute to this?

commented

Hey, thank you for offering! Sasquatch is awesome btw!

At an alpha level at least, this is basically done. We're just waiting for Bluesky team to turn on external federation. After they do that and we ship this, we'll discover a ton of bugs and missing things, as usual, but I'm not trying too hard to find all of those beforehand.

The one main thing I'm thinking about is #744, which currently caps us at 10k external accounts bridged into Bluesky. That sounds high, but I suspect it will arrive sooner than we expect.

Otherwise, feel free to look at the open issues and see if any of them grab you, I'd love the help!

commented

Oh, there's also #728, which somewhat blocks us from supporting reposts until bluesky-social/atproto#1811 is addressed. Feel free to evangelize or even look into fixing that!

commented

So I guess it's federating now.

commented

Just curious about the status of Bluesky support? Given this:

We're just waiting for Bluesky team to turn on external federation. After they do that and we ship this, we'll discover a ton of bugs and missing things, as usual, but I'm not trying too hard to find all of those beforehand.

Given that federation has been active over a month, I'm wondering when this will get activated. There are people on Bluesky I'd love to be able to follow from a Mastodon server (even passively, without fully functional federation), but this doesn't seem to be working yet, e.g. searching for @username.bsky.social@atproto.brid.gy from Mastodon doesn't return a profile for me to follow.

commented

@afontenot yes! Bluesky's federation is currently only in beta, limited to just 10 accounts per server, so we can't launch until they lift that limit.

We also have some work left to do here too, but that limit is probably still the main blocker right now.

commented

@snarfed thanks very much for the update!

commented

Regarding task

decide whether we have a BF protocol per lexicon/app, eg Bluesky, or just one for all of ATProto. We'd prefer the latter, but we need some app-specific semantics. Not sure yet if we can avoid all of those...

hasn't a decision been de-facto already made, at least for the time-being?

commented

Hmm! Surprisingly no, I don't think so. We only have the one atproto BF protocol right now, which hard codes the app.bsky lexicons and semantics. If/when we want to support new lexicons, like eg the new long-form writing service being built on ATProto (can't remember the name), I still don't know how we'd do it in BF. Fortunately we don't need to figure that out any time soon.

commented

the new long-form writing service being built on ATProto (can't remember the name)

     I don't think this is what you were referring to, but the owner/author of the blog hosted at snorre.io also built their site's comment system on top of BlueSky.

commented

Fun! Always nice to see other backfeed implementations like https://brid.gy/ (non-Fed). Agreed though, that's different, and it also still uses the existing app.bsky.* lexicons.

commented

Adding more todos and manual tests to the description at the top here, pinning this issue.

commented

Closing this! Initial implementation is complete. I'm now tracking ongoing bugs and updates in the now label.