The API file system


Web APIs form the backbone of many technologies today, but working with them can be difficult, particularly when attempting to unit test software. Often developers rely on sample data to test code that interacts with APIs. The data typically come from the file system, requiring two separate abstractions (disk I/O and network I/O) to perform the same task in different contexts. apifs aims to solve (or at least help with) this problem by providing a file system abstraction for HTTP requests and responses.

How it works

apifs uses a Go implementation of FUSE from the Bazil project. When the apifs service is run, apifs mounts itself at the given directory. Users create a directory or directory tree in apifs that parses to a valid HTTP URL - for example, /mnt/apifs/api.open-notify.org/astros.json - and initiate connections by reading clone files located in each subdirectory. By passing values to a control file for a given connection, requests are asynchronously initiated and the response is written to a file for the given connection.

Because all files on apifs are actual files as seen by the operating system, a developer can copy the whole directory tree elsewhere and capture the current state of an API for later testing. All HTTP requests and responses will simply be written to disk and made into static files.

OS X users will need to use OSXFUSE. Linux users will need to make sure they have permissions to use FUSE as a given user.

Challenges I ran into

File systems in general are complex and prone to many issues, including race conditions and permissions issues. While every reasonable attempt was made to mitigate these, realistically it is very difficult to write and thoroughly test a functioning file system in 48 hours.

Accomplishments that I'm proud of

It works! Implementing multiple connections for a given URL was difficult, but seeing actual HTTP requests and responses move through the file system was a reward in itself.

What I learned

File systems are a challenge to get right. Being able to have one unifying abstraction for different forms of I/O is very useful though, especially from a development and testing perspective.

What's next for apifs

This current implementation of apifs is largely a prototype. Future work includes more thorough testing of the file system itself, as well as support for cookies and output of other useful response data (status codes, MIME types, and so on).

Built with

  • go
  • fuse
  • bazil

Try it out