Loading...
Skip to main content

Fixie

Development environment setup​

Fixie uses a few tools to streamline development:

  • Homebrew is a package manager for MacOS that also mostly works for Linux. If you're running Debian or Ubuntu Linux, you can alternatively get by with apt.
  • Poetry manages our Python packages.
  • Buf manages our protos.
  • Just simplifies our shell workflows. It frequently functions as our interface to all the other tools.
  • npm powers our frontend.
  • PostgreSQL powers our databases.

Here's a quick guide to setting up your local development environment. It's written for MacOS (with Homebrew), but each tool's website has simple alternatives if you prefer apt.

  1. Install homebrew:

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

  2. Use pyenv to create a Python environment with Python 3.11:

    brew update
    brew install xz
    brew install pyenv
    pyenv init
    pyenv install 3.11
    pyenv global 3.11

    # Optional
    pyenv shell 3.11

  3. Install buf:

    brew install bufbuild/buf/buf

  4. Install the protoc-gen-openapi plugin, required by buf:

    brew install go
    go install github.com/google/gnostic/cmd/protoc-gen-openapi

    If you run into any issues here, make sure that $HOME/go/bin is in your PATH.

  5. Install PostgreSQL with the pgvector extension:

    brew install postgresql@14
    brew install pgvector
    brew services start postgresql@14

    To use apt instead, see the Postgres docs for setting up the pgdg APT repository and then run

    sudo apt install postgresql-15
    sudo apt install postgresql-15-pgvector

  6. Create your development databases:

    createdb
    psql postgres -c "CREATE EXTENSION vector;"
    psql < postgres_setup.sql

    You can re-run the command any time you'd like to drop all your data to get a fresh start.

  7. Install just:

    brew install just

  8. Install git lfs because all the images in the repo are hosted there :

    brew install git-lfs
    git lfs install
    git lfs pull

  9. Install Node.js:

    brew install node

  10. Set up the Public SDK:

    pip install fixieai
    fixie auth

  11. In this directory, run just install. If you have any issues with pip here try following these instructions https://pip.pypa.io/en/stable/installation/

    If you get any module error messages, run git submodule update --init. The error messages may look something like:

    fixie/chain/sql/evaluation/spider/metric/spider_exact_match.py:11: error: Module "third_party.elementai.spider" has no attribute "evaluation"  [attr-defined]
    fixie/chain/sql/evaluation/spider/metric/spider_test_suite.py:11: error: Module "third_party.elementai.test_suite_sql_eval" has no attribute "evaluation"  [attr-defined]
    Found 2 errors in 2 files (checked 233 source files)
    error: Recipe `install` failed on line 28 with exit code 1

Frontend​

If you're going to develop the frontend:

  1. Install Volta.
  2. Install pnpm.

Google Cloud credentials​

To access the relevant Google Cloud resources locally you will need allow Google SDKs to use your @fixie.ai Google Account.

  1. Install gcloud
  2. When it asks you which cloud project to use, choose fixie-frame.
  3. Run just setup-gcloud-auth

Formatting code​

We use the Black formatter, and mostly adhere to the Google Python style guide. Run just format to run black and isort on all of the code.

Running checks​

Run just check to run formatting and type checks.

Running tests​

Run just test to run tests.

Running a local copy of the server​

You need three running processes:

  1. Backend
  2. beta.fixie.ai frontend (only needed if you're making changes in src/frontend)
  3. Generic Sidekick Frontend UI (GSF)
  4. The Default Agent Runtime For Fixie (DARFF)

Run these proccees with:

  1. just run
  2. just frontend
  3. just frontend-sidekick-ui
  4. just default-agent-runtime-server

just run will produce a lot of output. You'll know the server is ready when the output includes:

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Run just browse to interact with the local app.

Running the console UI​

Run just console to run the console UI.

Adding new Python dependencies​

Add them to pyproject.toml and run poetry update && poetry install.

Development process​

When developing new code in this repository, please follow these steps:

  1. Create a branch named after your username, e.g., git checkout -b zach/new-feature
  2. Make the changes in your branch.
  3. Run just to ensure that the code is properly formatted and checks and tests pass.
  4. Use just run to test the server locally.
  5. When you're ready to request a code review, push your branch to GitHub: git push.
  6. Create a pull request for your changes, and add the relevant people as reviewers. Note that doing so will cause your branch to be deployed automatically to Google Cloud Run in its own hermetic environment, with its own Docker images and databases. You can see the process in the Actions tab of GitHub, and the resulting deployment will be linked from the "Conversation" page of your PR.
  7. Once your changes are approved, submit the code to main. This will cause the changes to be deployed to the main Fixie app URL.

Changing database models​

When editing a database model (typically in fixie/frame/models.py), a database migration will need to be created to ensure the production database can be updated in-place to know about the new field. To create a database migration, simply run just makemigrations, and you can update your local db instance via just migrate.

IMPORTANT: When changing a database model, the model edit and associated migration must be in their own PR (that is, separate from any other feature functionality). Reverting a migration can be problematic, and this approach allows us to ensure any database changes are low-risk.

Deployment on Google Cloud Run​

Our production and staging environments are deployed using Google Cloud Run. This requires building a Docker image containing the local code, along with configuration to cause the deployed service to use Google Cloud SQL as the database for storing user data.

These Docker images are built and deployed automatically when new code is pushed to the main branch, using GitHub Actions. The configuration for GitHub Actions are in the .github directory.

You can perform a manual build and push by following these steps:

  1. Run just push-docker to build and push the Docker images based on the code in your local branch.
  2. Run ./scripts/deploy-cloud-run.sh. This script takes two arguments. --environment-name is the name of the Cloud Run service to deploy, which is usually named after your local branch. --image-tag is the tag for the Docker image to deploy, which you get when running just push-docker step above.

Running Docker image locally​

You can also build and run Docker images locally, which is a good way to test the service in a hermetic environment without depending on your local machine configuration. To do this, run just run-docker. This will build the Docker images and run them locally. Note that the local Docker image does not depend on the Google Cloud SQL database; it uses a local SQLite database, so state is not persisted outside of the container.

Generating new public/private key pair​

$ openssl genpkey -algorithm ED25519 -out private.pem
$ openssl pkey -in private.pem -pubout -out public.pem
$ gcloud secrets versions add llamalabs-private-key --data-file private.pem
$ gcloud secrets versions add llamalabs-public-key --data-file public.pem
$ rm private.pem public.pem