From a0f198e89414259a4fd75871bf9b13d08ffcbd19 Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Wed, 4 Feb 2026 17:16:06 +0100 Subject: [PATCH] docs(01): research phase domain Phase 01: Foundation - Standard stack identified (Mill, ZIO HTTP, Quill, Flyway, Besom) - Architecture patterns documented (service layer, typed errors, HOCON config) - Pitfalls catalogued (effect wrapping, transaction nesting, Besom laziness) --- .planning/phases/01-foundation/01-RESEARCH.md | 510 ++++++++++++++++++ 1 file changed, 510 insertions(+) create mode 100644 .planning/phases/01-foundation/01-RESEARCH.md diff --git a/.planning/phases/01-foundation/01-RESEARCH.md b/.planning/phases/01-foundation/01-RESEARCH.md new file mode 100644 index 0000000..0464bb7 --- /dev/null +++ b/.planning/phases/01-foundation/01-RESEARCH.md @@ -0,0 +1,510 @@ +# Phase 1: Foundation - Research + +**Researched:** 2026-02-04 +**Domain:** Scala/ZIO stack with Mill build, PostgreSQL/Quill, and Pulumi deployment +**Confidence:** HIGH + +## Summary + +This phase establishes the foundational Scala/ZIO stack for SummerCMS. The research validates a modern, well-supported stack: Mill 1.1.x as build tool, ZIO HTTP 3.8.x for the HTTP server, Quill 4.8.x with quill-jdbc-zio for compile-time SQL validation against PostgreSQL, Flyway for database migrations (wrapped in ZIO effects), and Besom 0.5.x for Pulumi-based infrastructure as code. + +The stack is coherent and production-ready. Mill provides fast builds with YAML-based declarative configuration and fat JAR assembly. ZIO HTTP offers high-performance Netty-backed HTTP with native ZIO integration. Quill delivers compile-time SQL validation with PostgreSQL support. Flyway is the standard JVM migration tool with established ZIO wrappers. Besom brings Pulumi to Scala 3 with pure functional semantics. + +**Primary recommendation:** Use Mill 1.1.x with declarative `build.mill.yaml` for simplicity, ZIO HTTP 3.8.x with zio-config-typesafe for HOCON configuration, Quill 4.8.6 via quill-jdbc-zio for database access, Flyway wrapped in ZIO effects for migrations, and Besom 0.5.x with Scala CLI for Pulumi deployment. + +## Standard Stack + +The established libraries/tools for this domain: + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| Mill | 1.1.1 | Build tool | 3-7x faster than Maven/Gradle, declarative YAML config, native Scala support | +| ZIO | 2.1.x | Effect system | Industry-standard FP effect library for Scala, excellent ecosystem | +| ZIO HTTP | 3.8.1 | HTTP server | Netty-backed, native ZIO, high concurrency via fibers | +| Quill | 4.8.6 | Database queries | Compile-time SQL validation, type-safe, PostgreSQL support | +| Flyway | 10.x | Migrations | JVM standard, version tracking, rollback support | +| Besom | 0.5.0 | IaC (Pulumi) | Scala 3 native Pulumi SDK, functional semantics | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| zio-config | 4.0.5 | Configuration | HOCON file loading with environment overrides | +| zio-config-typesafe | 4.0.5 | HOCON support | Parse application.conf files | +| zio-config-magnolia | 4.0.2 | Config derivation | Auto-derive config from case classes | +| postgresql | 42.7.x | JDBC driver | PostgreSQL database connectivity | +| HikariCP | 5.x | Connection pooling | Built into quill-jdbc-zio | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| Mill | SBT | SBT has larger ecosystem but slower, more complex | +| Mill | Scala CLI | Scala CLI simpler for small projects, Mill better for larger ones | +| Quill | Doobie | Doobie uses SQL strings, Quill has compile-time validation | +| Flyway | Liquibase | Liquibase XML-heavy, Flyway simpler for SQL migrations | +| Besom | Pulumi Java SDK | Java SDK exists but Besom is Scala-native, better FP support | + +**Installation (Mill build.mill.yaml):** +```yaml +extends: ScalaModule +scalaVersion: 3.8.1 +mvnDeps: + - dev.zio::zio:2.1.14 + - dev.zio::zio-http:3.8.1 + - dev.zio::zio-config:4.0.5 + - dev.zio::zio-config-typesafe:4.0.5 + - dev.zio::zio-config-magnolia:4.0.2 + - io.getquill::quill-jdbc-zio:4.8.6 + - org.postgresql:postgresql:42.7.4 + - org.flywaydb:flyway-core:10.23.0 + - org.flywaydb:flyway-database-postgresql:10.23.0 +``` + +## Architecture Patterns + +### Recommended Project Structure +``` +summercms/ +├── build.mill.yaml # Mill declarative config +├── src/ +│ ├── Main.scala # Application entry point +│ ├── config/ +│ │ └── AppConfig.scala # Configuration case classes +│ ├── db/ +│ │ ├── Migrator.scala # Flyway wrapper in ZIO +│ │ └── QuillContext.scala # Quill PostgreSQL context +│ ├── repository/ +│ │ ├── Repository.scala # Base repository trait +│ │ └── UserRepository.scala # Example repository +│ ├── service/ +│ │ └── UserService.scala # Business logic layer +│ └── api/ +│ ├── Routes.scala # HTTP route definitions +│ └── HealthRoutes.scala # Health check endpoints +├── resources/ +│ ├── application.conf # HOCON configuration +│ └── db/migration/ # Flyway SQL migrations +├── test/ +│ └── src/ # Test sources +└── infra/ # Pulumi/Besom infrastructure + ├── project.scala # Scala CLI project config + └── Main.scala # Infrastructure definition +``` + +### Pattern 1: ZIO Service Layer (Trait + Live) +**What:** Define services as traits with ZLayer-based implementations +**When to use:** All service components (repositories, business services) +**Example:** +```scala +// Source: https://softwaremill.com/structuring-zio-2-applications/ +// Trait defines the contract +trait UserRepository: + def findById(id: Long): IO[RepositoryError, Option[User]] + def create(user: User): IO[RepositoryError, User] + def update(user: User): IO[RepositoryError, User] + def delete(id: Long): IO[RepositoryError, Unit] + +// Companion object provides ZLayer +object UserRepository: + val live: ZLayer[Quill.Postgres[SnakeCase], Nothing, UserRepository] = + ZLayer.fromFunction(UserRepositoryLive(_)) + +// Implementation class +class UserRepositoryLive(quill: Quill.Postgres[SnakeCase]) extends UserRepository: + import quill.* + + def findById(id: Long): IO[RepositoryError, Option[User]] = + run(query[User].filter(_.id == lift(id))) + .map(_.headOption) + .mapError(e => RepositoryError.DatabaseError(e.getMessage)) +``` + +### Pattern 2: Typed Error ADT +**What:** Define domain errors as sealed traits for exhaustive handling +**When to use:** Repository and service error types +**Example:** +```scala +// Source: https://zio.dev/reference/error-management/best-practices/unexpected-errors/ +sealed trait RepositoryError +object RepositoryError: + case class NotFound(entity: String, id: Long) extends RepositoryError + case class Conflict(entity: String, message: String) extends RepositoryError + case class ValidationError(errors: List[String]) extends RepositoryError + case class DatabaseError(message: String) extends RepositoryError + +// Use refineOrDie for unexpected errors +def findById(id: Long): IO[RepositoryError, Option[User]] = + run(query[User].filter(_.id == lift(id))) + .map(_.headOption) + .refineOrDie { + case e: SQLException => RepositoryError.DatabaseError(e.getMessage) + } +``` + +### Pattern 3: HOCON Configuration with ZIO Config +**What:** Type-safe configuration from HOCON files with environment overrides +**When to use:** Application configuration (server, database, etc.) +**Example:** +```scala +// Source: https://ziohttp.com/guides/integration-with-zio-config +// Config case class +case class AppConfig( + server: ServerConfig, + database: DatabaseConfig +) +case class ServerConfig(host: String, port: Int) +case class DatabaseConfig( + host: String, + port: Int, + database: String, + user: String, + password: String +) + +// application.conf +// server { +// host = "0.0.0.0" +// host = ${?SERVER_HOST} +// port = 8080 +// port = ${?SERVER_PORT} +// } + +// Bootstrap with config provider +object Main extends ZIOAppDefault: + override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] = + Runtime.setConfigProvider(ConfigProvider.fromResourcePath()) +``` + +### Pattern 4: Quill PostgreSQL Context with ZIO +**What:** Configure Quill for compile-time validated PostgreSQL queries +**When to use:** All database access +**Example:** +```scala +// Source: https://zio.dev/zio-quill/getting-started/ +import io.getquill.* +import io.getquill.jdbczio.Quill + +// Create the Quill layer +val quillLayer: ZLayer[DataSource, Nothing, Quill.Postgres[SnakeCase]] = + Quill.Postgres.fromNamingStrategy(SnakeCase) + +// DataSource from config +val dataSourceLayer: ZLayer[Any, Throwable, DataSource] = + Quill.DataSource.fromPrefix("database") + +// Combine layers +val dbLayer = dataSourceLayer >>> quillLayer +``` + +### Pattern 5: Flyway Migrations Wrapped in ZIO +**What:** Run database migrations as ZIO effects +**When to use:** Application startup, CLI commands +**Example:** +```scala +// Source: https://github.com/DenisNovac/zio-flyway-db-migrator (pattern) +import org.flywaydb.core.Flyway +import zio.* + +trait Migrator: + def migrate: Task[Int] + def rollback: Task[Unit] + def status: Task[MigrationStatus] + +object Migrator: + val live: ZLayer[DataSource, Nothing, Migrator] = + ZLayer.fromFunction { (ds: DataSource) => + new Migrator: + private val flyway = Flyway.configure() + .dataSource(ds) + .locations("classpath:db/migration") + .table("summer_migrations") + .load() + + def migrate: Task[Int] = ZIO.attempt(flyway.migrate().migrationsExecuted) + def rollback: Task[Unit] = ZIO.attempt(flyway.undo()).unit + def status: Task[MigrationStatus] = ZIO.attempt { + val info = flyway.info() + MigrationStatus( + current = Option(info.current()).map(_.getVersion.toString), + pending = info.pending().length + ) + } + } +``` + +### Anti-Patterns to Avoid +- **Using `ZIO.effect` for Futures:** Use `ZIO.fromFuture` instead to properly await async completion +- **Discarding effects in for-comprehensions:** Always chain effects with `flatMap` or `*>` +- **Double-wrapping ZIO in pattern matches:** Evaluate each case branch separately +- **Using `println` instead of ZIO Console:** Use `Console.printLine` for proper effect tracking +- **Returning ZIO inside yield:** The yield should return values, not effects +- **Typing unexpected errors:** Use `refineOrDie` to only type recoverable errors + +## Don't Hand-Roll + +Problems that look simple but have existing solutions: + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| SQL query building | String concatenation | Quill QDSL | SQL injection, type safety, compile-time validation | +| Database migrations | Custom version tracking | Flyway | Checksums, rollbacks, history table, team coordination | +| Configuration loading | Manual parsing | zio-config-typesafe | Environment overrides, nested configs, validation | +| Connection pooling | Manual pool management | HikariCP (via Quill) | Connection lifecycle, leak detection, metrics | +| HTTP routing | Manual path matching | ZIO HTTP Routes | Type-safe path params, method matching, middleware | +| Error handling | Try/catch | ZIO typed errors | Composability, resource safety, stack traces | +| Dependency injection | Manual wiring | ZLayer | Compile-time verification, resource management | +| Fat JAR building | Manual manifest/classpath | Mill assembly | Conflict resolution, main class detection | +| Infrastructure code | CloudFormation/Terraform | Besom (Pulumi) | Type safety, Scala ecosystem, refactoring support | + +**Key insight:** The ZIO ecosystem provides cohesive solutions that work together. Hand-rolling any component breaks the effect composition model and loses compile-time guarantees. + +## Common Pitfalls + +### Pitfall 1: Quill Variable Name Conflict +**What goes wrong:** Naming a variable `io` causes compiler errors with Quill +**Why it happens:** Quill's package is `io.getquill`, causing naming collision +**How to avoid:** Never use `io` as a variable name in files importing Quill +**Warning signs:** Strange "not found: value io" errors after adding Quill + +### Pitfall 2: Wrong ZIO Effect Wrapping for Futures +**What goes wrong:** Async operations complete prematurely or run on wrong thread pool +**Why it happens:** Using `ZIO.attempt` instead of `ZIO.fromFuture` for Future-returning methods +**How to avoid:** Always use `ZIO.fromFuture` for methods returning `Future[A]` +**Warning signs:** Unexpected null values, race conditions, operations not completing + +### Pitfall 3: Nested Transaction Failures in Quill +**What goes wrong:** Foreign key constraint violations in nested transactions +**Why it happens:** PostgreSQL async driver starts separate transactions for nested calls +**How to avoid:** Flatten transaction logic, avoid nesting `ctx.transaction` blocks +**Warning signs:** FK constraint errors that work in simple tests but fail in complex flows + +### Pitfall 4: Missing Effect Chaining in For-Comprehensions +**What goes wrong:** Effects are silently discarded, side effects don't execute +**Why it happens:** Not using `<-` or `*>` to chain effects, just sequencing statements +**How to avoid:** Use `_ <- effect` or `effect1 *> effect2` for side-effect-only operations +**Warning signs:** Logging statements not appearing, database writes not persisting + +### Pitfall 5: Besom Resource Laziness +**What goes wrong:** Cloud resources not deployed despite being defined +**Why it happens:** Besom resources are lazy and must be referenced to deploy +**How to avoid:** Always export or reference resources in Stack.exports or other resources +**Warning signs:** `pulumi up` shows no changes when resources are defined + +### Pitfall 6: Mill Not Supported by Besom +**What goes wrong:** Cannot use Mill for Pulumi infrastructure projects +**Why it happens:** Besom only supports Scala CLI, SBT, Maven, Gradle (not Mill) +**How to avoid:** Use Scala CLI for the `infra/` directory, Mill for main application +**Warning signs:** Besom compilation failures when using Mill + +### Pitfall 7: GraalVM Native Image Reflection Issues +**What goes wrong:** Runtime errors in native image builds +**Why it happens:** Native image requires reflection configuration for frameworks +**How to avoid:** Start with fat JAR, only optimize to native image when needed +**Warning signs:** ClassNotFoundException, NoSuchMethodError at runtime in native builds + +## Code Examples + +Verified patterns from official sources: + +### Basic ZIO HTTP Server +```scala +// Source: https://ziohttp.com/ +import zio.* +import zio.http.* + +object Main extends ZIOAppDefault: + val routes = Routes( + Method.GET / "health" -> handler(Response.text("ok")), + Method.GET / "ready" -> handler { + // Check database connectivity + ZIO.serviceWithZIO[DataSource](ds => + ZIO.attempt(ds.getConnection.close()) + ).as(Response.text("ready")) + .catchAll(_ => ZIO.succeed(Response.status(Status.ServiceUnavailable))) + } + ) + + def run = Server.serve(routes).provide( + Server.defaultWithPort(8080), + dataSourceLayer + ) +``` + +### Quill PostgreSQL Query +```scala +// Source: https://zio.dev/zio-quill/getting-started/ +import io.getquill.* +import io.getquill.jdbczio.Quill + +case class User(id: Long, email: String, createdAt: java.time.Instant) + +class UserRepositoryLive(quill: Quill.Postgres[SnakeCase]) extends UserRepository: + import quill.* + + inline def users = quote(querySchema[User]("summer_users")) + + def findById(id: Long): Task[Option[User]] = + run(users.filter(_.id == lift(id))).map(_.headOption) + + def create(user: User): Task[User] = + run(users.insertValue(lift(user)).returning(u => u)) + + def findByEmail(email: String): Task[Option[User]] = + run(users.filter(_.email == lift(email))).map(_.headOption) +``` + +### HOCON Configuration +```hocon +# Source: https://ziohttp.com/guides/integration-with-zio-config +# resources/application.conf + +server { + host = "0.0.0.0" + host = ${?SERVER_HOST} + port = 8080 + port = ${?SERVER_PORT} +} + +database { + dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource" + dataSource { + serverName = "localhost" + serverName = ${?DB_HOST} + portNumber = 5432 + portNumber = ${?DB_PORT} + databaseName = "summercms" + databaseName = ${?DB_NAME} + user = "summercms" + user = ${?DB_USER} + password = "summercms" + password = ${?DB_PASSWORD} + } + connectionTimeout = 30000 +} +``` + +### Mill Assembly Configuration +```scala +// Source: https://mill-build.org/mill/scalalib/intro.html +// build.mill (for programmatic config with assembly) +package build +import mill.*, scalalib.*, scalalib.assembly.* + +object summercms extends ScalaModule: + def scalaVersion = "3.8.1" + def mvnDeps = Seq( + mvn"dev.zio::zio:2.1.14", + mvn"dev.zio::zio-http:3.8.1", + // ... other deps + ) + + // Fat JAR configuration + def assemblyRules = Seq( + Rule.Relocate("META-INF/services/**" -> "META-INF/services/@1"), + Rule.ExcludePattern("META-INF/*.SF"), + Rule.ExcludePattern("META-INF/*.DSA"), + Rule.ExcludePattern("META-INF/*.RSA") + ) +``` + +### Besom AWS Infrastructure +```scala +// Source: https://virtuslab.github.io/besom/docs/getting_started/ +// infra/Main.scala (using Scala CLI) +//> using scala 3.3.1 +//> using dep org.virtuslab::besom-core:0.5.0 +//> using dep org.virtuslab::besom-aws:6.66.0 + +import besom.* +import besom.api.aws + +@main def main = Pulumi.run { + val bucket = aws.s3.Bucket("summercms-assets") + + val db = aws.rds.Instance("summercms-db", + aws.rds.InstanceArgs( + engine = "postgres", + engineVersion = "16.4", + instanceClass = "db.t3.micro", + allocatedStorage = 20, + dbName = "summercms", + username = "summercms", + password = config.requireSecret("dbPassword"), + skipFinalSnapshot = true + ) + ) + + Stack.exports( + bucketName = bucket.bucket, + dbEndpoint = db.endpoint + ) +} +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| SBT builds | Mill with YAML config | Mill 1.1.0 (Jan 2026) | 3-7x faster builds, simpler config | +| Quill Scala 2 | ProtoQuill (Scala 3) | Quill 4.x | Better compile-time metaprogramming | +| ZIO 1.x service pattern | ZIO 2 ZLayer.make | ZIO 2.0 (2022) | Simpler layer composition | +| Manual Pulumi Java | Besom Scala SDK | Besom 0.1.0 (2024) | Native Scala 3 IaC | +| Thread-local transactions | ZIO effect-based transactions | quill-jdbc-zio | No thread-local pollution | + +**Deprecated/outdated:** +- `io.d11` organization for ZIO HTTP - now under `dev.zio` +- Quill `Connection` dependency - now uses `DataSource` +- `ZIO.effect` - renamed to `ZIO.attempt` in ZIO 2 +- `ZManaged` - replaced by `ZIO.scoped` in ZIO 2 + +## Open Questions + +Things that couldn't be fully resolved: + +1. **Besom + Mill Integration** + - What we know: Besom officially only supports Scala CLI, SBT, Maven, Gradle + - What's unclear: Whether Mill could work with custom configuration + - Recommendation: Use Scala CLI for `infra/` directory (separate from main Mill build) + +2. **Hot-Reload in Development** + - What we know: Mill has `-w` watch mode, but that recompiles and restarts + - What's unclear: True hot-reload without restart for Scala/ZIO + - Recommendation: Use Mill `-w` for now; investigate JRebel or ZIO Test live reload for future + +3. **Quill Scala 3 Maturity** + - What we know: ProtoQuill exists for Scala 3, supports ZIO contexts + - What's unclear: Feature parity with Scala 2 Quill, edge case stability + - Recommendation: Use quill-jdbc-zio 4.8.6 which supports Scala 3, monitor issues + +4. **Transaction Isolation Levels** + - What we know: Quill wraps transactions, PostgreSQL supports isolation levels + - What's unclear: How to set isolation level per-transaction in quill-jdbc-zio + - Recommendation: Use default (READ COMMITTED), document if customization needed + +## Sources + +### Primary (HIGH confidence) +- [Mill Build Tool Documentation](https://mill-build.org/mill/index.html) - Build configuration, assembly, watch mode +- [ZIO HTTP Documentation](https://ziohttp.com/) - Server setup, routes, configuration +- [ZIO Quill Documentation](https://zio.dev/zio-quill/) - Query DSL, contexts, transactions +- [Besom Documentation](https://virtuslab.github.io/besom/) - Pulumi Scala SDK setup +- [ZIO Error Management](https://zio.dev/reference/error-management/best-practices/) - Typed errors, refineOrDie + +### Secondary (MEDIUM confidence) +- [SoftwareMill ZIO 2 Structure](https://softwaremill.com/structuring-zio-2-applications/) - Service layer patterns +- [ZIO HTTP GitHub Releases](https://github.com/zio/zio-http/releases) - Version 3.8.1 verified +- [Mill Native Image Plugin](https://github.com/alexarchambault/mill-native-image) - GraalVM integration + +### Tertiary (LOW confidence) +- [Wix ZIO Pitfalls](https://medium.com/wix-engineering/5-pitfalls-to-avoid-when-starting-to-work-with-zio-adefdc7d2d5c) - Common mistakes (older article, patterns still valid) +- [zio-flyway-db-migrator](https://github.com/DenisNovac/zio-flyway-db-migrator) - Migration pattern example + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - All libraries verified via official docs and release pages +- Architecture: HIGH - Patterns from official ZIO documentation and SoftwareMill +- Pitfalls: MEDIUM - Mix of official docs and community experience posts + +**Research date:** 2026-02-04 +**Valid until:** 2026-03-04 (30 days - stable ecosystem)