Semptic/go-crowley
Watch
1
0
Fork
You've already forked go-crowley
 0
mirror of https://github.com/Semptic/go-crowley.git synced 2026-10-18 12:21:01 +00:00
Code Issues Projects Packages Wiki Activity
POC to use go as a webcrawler with postgres as queue
13 commits 1 branch 0 tags 49 KiB
  • Go 82.5%
  • templ 17.3%
  • Shell 0.2%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Stefan Kuhn 9bc58f33d6
 Document how to start postgres
2024-02-27 21:08:54 +01:00
cmd Reformat 2024-02-27 20:43:27 +01:00
db/migrations I'm BATMAN 2024-02-27 20:17:12 +01:00
.env I'm BATMAN 2024-02-27 20:17:12 +01:00
.gitignore Add gitignore file 2024-02-27 20:17:45 +01:00
.monitor.air.toml I'm BATMAN 2024-02-27 20:17:12 +01:00
docker-compose.yaml I'm BATMAN 2024-02-27 20:17:12 +01:00
go.mod I'm BATMAN 2024-02-27 20:17:12 +01:00
go.sum I'm BATMAN 2024-02-27 20:17:12 +01:00
LICENSE Initial commit 2024-02-27 20:35:15 +01:00
README.md Document how to start postgres 2024-02-27 21:08:54 +01:00
watch_monitor.sh I'm BATMAN 2024-02-27 20:17:12 +01:00

README.md

Crowley

Run

  1. Start postgres
docker compose up -d
  1. Add couple of domains to crawl
go run cmd/manager/*.go -p wiki -u https://en.wikipedia.org/wiki/Music_genre
go run cmd/manager/*.go -p hackernews -u https://news.ycombinator.com/
  1. Start the monitor
./watch_monitor.sh

or

go run cmd/monitor/*.go
  1. Start some worker (run the following in multiple terminals)
go run cmd/worker/*.go

Setup

Prerequisites

Install go, templ, air (Until this is merged install git clone -b feat-live-proxy https://github.com/ndajr/air && cd air && go install .).

DB

Use migrate for handling database migrations and pgx for postgres.

go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

Create new migration file

migrate create -dir db/migrations -ext sql [NAME]

Drop database

USER=<user>; PASS=<pass> migrate -database "postgres://$USER:$PASS@localhost:5432/galaxy?sslmode=disable" drop