docs(01): research phase domain

Phase 01: Foundation - Standard stack identified (Mill, ZIO HTTP, Quill, Flyway, Besom) - Architecture patterns documented (service layer, typed errors, HOCON config) - Pitfalls catalogued (effect wrapping, transaction nesting, Besom laziness)
2026-02-04 17:16:06 +01:00
parent 02ee2b296b
commit a0f198e894
1 changed files with 510 additions and 0 deletions
--- a/.planning/phases/01-foundation/01-RESEARCH.md
+++ b/.planning/phases/01-foundation/01-RESEARCH.md
@@ -0,0 +1,510 @@
+# Phase 1: Foundation - Research
+
+**Researched:** 2026-02-04
+**Domain:** Scala/ZIO stack with Mill build, PostgreSQL/Quill, and Pulumi deployment
+**Confidence:** HIGH
+
+## Summary
+
+This phase establishes the foundational Scala/ZIO stack for SummerCMS. The research validates a modern, well-supported stack: Mill 1.1.x as build tool, ZIO HTTP 3.8.x for the HTTP server, Quill 4.8.x with quill-jdbc-zio for compile-time SQL validation against PostgreSQL, Flyway for database migrations (wrapped in ZIO effects), and Besom 0.5.x for Pulumi-based infrastructure as code.
+
+The stack is coherent and production-ready. Mill provides fast builds with YAML-based declarative configuration and fat JAR assembly. ZIO HTTP offers high-performance Netty-backed HTTP with native ZIO integration. Quill delivers compile-time SQL validation with PostgreSQL support. Flyway is the standard JVM migration tool with established ZIO wrappers. Besom brings Pulumi to Scala 3 with pure functional semantics.
+
+**Primary recommendation:** Use Mill 1.1.x with declarative `build.mill.yaml` for simplicity, ZIO HTTP 3.8.x with zio-config-typesafe for HOCON configuration, Quill 4.8.6 via quill-jdbc-zio for database access, Flyway wrapped in ZIO effects for migrations, and Besom 0.5.x with Scala CLI for Pulumi deployment.
+
+## Standard Stack
+
+The established libraries/tools for this domain:
+
+### Core
+| Library | Version | Purpose | Why Standard |
+|---------|---------|---------|--------------|
+| Mill | 1.1.1 | Build tool | 3-7x faster than Maven/Gradle, declarative YAML config, native Scala support |
+| ZIO | 2.1.x | Effect system | Industry-standard FP effect library for Scala, excellent ecosystem |
+| ZIO HTTP | 3.8.1 | HTTP server | Netty-backed, native ZIO, high concurrency via fibers |
+| Quill | 4.8.6 | Database queries | Compile-time SQL validation, type-safe, PostgreSQL support |
+| Flyway | 10.x | Migrations | JVM standard, version tracking, rollback support |
+| Besom | 0.5.0 | IaC (Pulumi) | Scala 3 native Pulumi SDK, functional semantics |
+
+### Supporting
+| Library | Version | Purpose | When to Use |
+|---------|---------|---------|-------------|
+| zio-config | 4.0.5 | Configuration | HOCON file loading with environment overrides |
+| zio-config-typesafe | 4.0.5 | HOCON support | Parse application.conf files |
+| zio-config-magnolia | 4.0.2 | Config derivation | Auto-derive config from case classes |
+| postgresql | 42.7.x | JDBC driver | PostgreSQL database connectivity |
+| HikariCP | 5.x | Connection pooling | Built into quill-jdbc-zio |
+
+### Alternatives Considered
+| Instead of | Could Use | Tradeoff |
+|------------|-----------|----------|
+| Mill | SBT | SBT has larger ecosystem but slower, more complex |
+| Mill | Scala CLI | Scala CLI simpler for small projects, Mill better for larger ones |
+| Quill | Doobie | Doobie uses SQL strings, Quill has compile-time validation |
+| Flyway | Liquibase | Liquibase XML-heavy, Flyway simpler for SQL migrations |
+| Besom | Pulumi Java SDK | Java SDK exists but Besom is Scala-native, better FP support |
+
+**Installation (Mill build.mill.yaml):**
+```yaml
+extends: ScalaModule
+scalaVersion: 3.8.1
+mvnDeps:
+  - dev.zio::zio:2.1.14
+  - dev.zio::zio-http:3.8.1
+  - dev.zio::zio-config:4.0.5
+  - dev.zio::zio-config-typesafe:4.0.5
+  - dev.zio::zio-config-magnolia:4.0.2
+  - io.getquill::quill-jdbc-zio:4.8.6
+  - org.postgresql:postgresql:42.7.4
+  - org.flywaydb:flyway-core:10.23.0
+  - org.flywaydb:flyway-database-postgresql:10.23.0
+```
+
+## Architecture Patterns
+
+### Recommended Project Structure
+```
+summercms/
+├── build.mill.yaml           # Mill declarative config
+├── src/
+│   ├── Main.scala            # Application entry point
+│   ├── config/
+│   │   └── AppConfig.scala   # Configuration case classes
+│   ├── db/
+│   │   ├── Migrator.scala    # Flyway wrapper in ZIO
+│   │   └── QuillContext.scala # Quill PostgreSQL context
+│   ├── repository/
+│   │   ├── Repository.scala  # Base repository trait
+│   │   └── UserRepository.scala # Example repository
+│   ├── service/
+│   │   └── UserService.scala # Business logic layer
+│   └── api/
+│       ├── Routes.scala      # HTTP route definitions
+│       └── HealthRoutes.scala # Health check endpoints
+├── resources/
+│   ├── application.conf      # HOCON configuration
+│   └── db/migration/         # Flyway SQL migrations
+├── test/
+│   └── src/                  # Test sources
+└── infra/                    # Pulumi/Besom infrastructure
+    ├── project.scala         # Scala CLI project config
+    └── Main.scala            # Infrastructure definition
+```
+
+### Pattern 1: ZIO Service Layer (Trait + Live)
+**What:** Define services as traits with ZLayer-based implementations
+**When to use:** All service components (repositories, business services)
+**Example:**
+```scala
+// Source: https://softwaremill.com/structuring-zio-2-applications/
+// Trait defines the contract
+trait UserRepository:
+  def findById(id: Long): IO[RepositoryError, Option[User]]
+  def create(user: User): IO[RepositoryError, User]
+  def update(user: User): IO[RepositoryError, User]
+  def delete(id: Long): IO[RepositoryError, Unit]
+
+// Companion object provides ZLayer
+object UserRepository:
+  val live: ZLayer[Quill.Postgres[SnakeCase], Nothing, UserRepository] =
+    ZLayer.fromFunction(UserRepositoryLive(_))
+
+// Implementation class
+class UserRepositoryLive(quill: Quill.Postgres[SnakeCase]) extends UserRepository:
+  import quill.*
+
+  def findById(id: Long): IO[RepositoryError, Option[User]] =
+    run(query[User].filter(_.id == lift(id)))
+      .map(_.headOption)
+      .mapError(e => RepositoryError.DatabaseError(e.getMessage))
+```
+
+### Pattern 2: Typed Error ADT
+**What:** Define domain errors as sealed traits for exhaustive handling
+**When to use:** Repository and service error types
+**Example:**
+```scala
+// Source: https://zio.dev/reference/error-management/best-practices/unexpected-errors/
+sealed trait RepositoryError
+object RepositoryError:
+  case class NotFound(entity: String, id: Long) extends RepositoryError
+  case class Conflict(entity: String, message: String) extends RepositoryError
+  case class ValidationError(errors: List[String]) extends RepositoryError
+  case class DatabaseError(message: String) extends RepositoryError
+
+// Use refineOrDie for unexpected errors
+def findById(id: Long): IO[RepositoryError, Option[User]] =
+  run(query[User].filter(_.id == lift(id)))
+    .map(_.headOption)
+    .refineOrDie {
+      case e: SQLException => RepositoryError.DatabaseError(e.getMessage)
+    }
+```
+
+### Pattern 3: HOCON Configuration with ZIO Config
+**What:** Type-safe configuration from HOCON files with environment overrides
+**When to use:** Application configuration (server, database, etc.)
+**Example:**
+```scala
+// Source: https://ziohttp.com/guides/integration-with-zio-config
+// Config case class
+case class AppConfig(
+  server: ServerConfig,
+  database: DatabaseConfig
+)
+case class ServerConfig(host: String, port: Int)
+case class DatabaseConfig(
+  host: String,
+  port: Int,
+  database: String,
+  user: String,
+  password: String
+)
+
+// application.conf
+// server {
+//   host = "0.0.0.0"
+//   host = ${?SERVER_HOST}
+//   port = 8080
+//   port = ${?SERVER_PORT}
+// }
+
+// Bootstrap with config provider
+object Main extends ZIOAppDefault:
+  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
+    Runtime.setConfigProvider(ConfigProvider.fromResourcePath())
+```
+
+### Pattern 4: Quill PostgreSQL Context with ZIO
+**What:** Configure Quill for compile-time validated PostgreSQL queries
+**When to use:** All database access
+**Example:**
+```scala
+// Source: https://zio.dev/zio-quill/getting-started/
+import io.getquill.*
+import io.getquill.jdbczio.Quill
+
+// Create the Quill layer
+val quillLayer: ZLayer[DataSource, Nothing, Quill.Postgres[SnakeCase]] =
+  Quill.Postgres.fromNamingStrategy(SnakeCase)
+
+// DataSource from config
+val dataSourceLayer: ZLayer[Any, Throwable, DataSource] =
+  Quill.DataSource.fromPrefix("database")
+
+// Combine layers
+val dbLayer = dataSourceLayer >>> quillLayer
+```
+
+### Pattern 5: Flyway Migrations Wrapped in ZIO
+**What:** Run database migrations as ZIO effects
+**When to use:** Application startup, CLI commands
+**Example:**
+```scala
+// Source: https://github.com/DenisNovac/zio-flyway-db-migrator (pattern)
+import org.flywaydb.core.Flyway
+import zio.*
+
+trait Migrator:
+  def migrate: Task[Int]
+  def rollback: Task[Unit]
+  def status: Task[MigrationStatus]
+
+object Migrator:
+  val live: ZLayer[DataSource, Nothing, Migrator] =
+    ZLayer.fromFunction { (ds: DataSource) =>
+      new Migrator:
+        private val flyway = Flyway.configure()
+          .dataSource(ds)
+          .locations("classpath:db/migration")
+          .table("summer_migrations")
+          .load()
+
+        def migrate: Task[Int] = ZIO.attempt(flyway.migrate().migrationsExecuted)
+        def rollback: Task[Unit] = ZIO.attempt(flyway.undo()).unit
+        def status: Task[MigrationStatus] = ZIO.attempt {
+          val info = flyway.info()
+          MigrationStatus(
+            current = Option(info.current()).map(_.getVersion.toString),
+            pending = info.pending().length
+          )
+        }
+    }
+```
+
+### Anti-Patterns to Avoid
+- **Using `ZIO.effect` for Futures:** Use `ZIO.fromFuture` instead to properly await async completion
+- **Discarding effects in for-comprehensions:** Always chain effects with `flatMap` or `*>`
+- **Double-wrapping ZIO in pattern matches:** Evaluate each case branch separately
+- **Using `println` instead of ZIO Console:** Use `Console.printLine` for proper effect tracking
+- **Returning ZIO inside yield:** The yield should return values, not effects
+- **Typing unexpected errors:** Use `refineOrDie` to only type recoverable errors
+
+## Don't Hand-Roll
+
+Problems that look simple but have existing solutions:
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| SQL query building | String concatenation | Quill QDSL | SQL injection, type safety, compile-time validation |
+| Database migrations | Custom version tracking | Flyway | Checksums, rollbacks, history table, team coordination |
+| Configuration loading | Manual parsing | zio-config-typesafe | Environment overrides, nested configs, validation |
+| Connection pooling | Manual pool management | HikariCP (via Quill) | Connection lifecycle, leak detection, metrics |
+| HTTP routing | Manual path matching | ZIO HTTP Routes | Type-safe path params, method matching, middleware |
+| Error handling | Try/catch | ZIO typed errors | Composability, resource safety, stack traces |
+| Dependency injection | Manual wiring | ZLayer | Compile-time verification, resource management |
+| Fat JAR building | Manual manifest/classpath | Mill assembly | Conflict resolution, main class detection |
+| Infrastructure code | CloudFormation/Terraform | Besom (Pulumi) | Type safety, Scala ecosystem, refactoring support |
+
+**Key insight:** The ZIO ecosystem provides cohesive solutions that work together. Hand-rolling any component breaks the effect composition model and loses compile-time guarantees.
+
+## Common Pitfalls
+
+### Pitfall 1: Quill Variable Name Conflict
+**What goes wrong:** Naming a variable `io` causes compiler errors with Quill
+**Why it happens:** Quill's package is `io.getquill`, causing naming collision
+**How to avoid:** Never use `io` as a variable name in files importing Quill
+**Warning signs:** Strange "not found: value io" errors after adding Quill
+
+### Pitfall 2: Wrong ZIO Effect Wrapping for Futures
+**What goes wrong:** Async operations complete prematurely or run on wrong thread pool
+**Why it happens:** Using `ZIO.attempt` instead of `ZIO.fromFuture` for Future-returning methods
+**How to avoid:** Always use `ZIO.fromFuture` for methods returning `Future[A]`
+**Warning signs:** Unexpected null values, race conditions, operations not completing
+
+### Pitfall 3: Nested Transaction Failures in Quill
+**What goes wrong:** Foreign key constraint violations in nested transactions
+**Why it happens:** PostgreSQL async driver starts separate transactions for nested calls
+**How to avoid:** Flatten transaction logic, avoid nesting `ctx.transaction` blocks
+**Warning signs:** FK constraint errors that work in simple tests but fail in complex flows
+
+### Pitfall 4: Missing Effect Chaining in For-Comprehensions
+**What goes wrong:** Effects are silently discarded, side effects don't execute
+**Why it happens:** Not using `<-` or `*>` to chain effects, just sequencing statements
+**How to avoid:** Use `_ <- effect` or `effect1 *> effect2` for side-effect-only operations
+**Warning signs:** Logging statements not appearing, database writes not persisting
+
+### Pitfall 5: Besom Resource Laziness
+**What goes wrong:** Cloud resources not deployed despite being defined
+**Why it happens:** Besom resources are lazy and must be referenced to deploy
+**How to avoid:** Always export or reference resources in Stack.exports or other resources
+**Warning signs:** `pulumi up` shows no changes when resources are defined
+
+### Pitfall 6: Mill Not Supported by Besom
+**What goes wrong:** Cannot use Mill for Pulumi infrastructure projects
+**Why it happens:** Besom only supports Scala CLI, SBT, Maven, Gradle (not Mill)
+**How to avoid:** Use Scala CLI for the `infra/` directory, Mill for main application
+**Warning signs:** Besom compilation failures when using Mill
+
+### Pitfall 7: GraalVM Native Image Reflection Issues
+**What goes wrong:** Runtime errors in native image builds
+**Why it happens:** Native image requires reflection configuration for frameworks
+**How to avoid:** Start with fat JAR, only optimize to native image when needed
+**Warning signs:** ClassNotFoundException, NoSuchMethodError at runtime in native builds
+
+## Code Examples
+
+Verified patterns from official sources:
+
+### Basic ZIO HTTP Server
+```scala
+// Source: https://ziohttp.com/
+import zio.*
+import zio.http.*
+
+object Main extends ZIOAppDefault:
+  val routes = Routes(
+    Method.GET / "health" -> handler(Response.text("ok")),
+    Method.GET / "ready" -> handler {
+      // Check database connectivity
+      ZIO.serviceWithZIO[DataSource](ds =>
+        ZIO.attempt(ds.getConnection.close())
+      ).as(Response.text("ready"))
+       .catchAll(_ => ZIO.succeed(Response.status(Status.ServiceUnavailable)))
+    }
+  )
+
+  def run = Server.serve(routes).provide(
+    Server.defaultWithPort(8080),
+    dataSourceLayer
+  )
+```
+
+### Quill PostgreSQL Query
+```scala
+// Source: https://zio.dev/zio-quill/getting-started/
+import io.getquill.*
+import io.getquill.jdbczio.Quill
+
+case class User(id: Long, email: String, createdAt: java.time.Instant)
+
+class UserRepositoryLive(quill: Quill.Postgres[SnakeCase]) extends UserRepository:
+  import quill.*
+
+  inline def users = quote(querySchema[User]("summer_users"))
+
+  def findById(id: Long): Task[Option[User]] =
+    run(users.filter(_.id == lift(id))).map(_.headOption)
+
+  def create(user: User): Task[User] =
+    run(users.insertValue(lift(user)).returning(u => u))
+
+  def findByEmail(email: String): Task[Option[User]] =
+    run(users.filter(_.email == lift(email))).map(_.headOption)
+```
+
+### HOCON Configuration
+```hocon
+# Source: https://ziohttp.com/guides/integration-with-zio-config
+# resources/application.conf
+
+server {
+  host = "0.0.0.0"
+  host = ${?SERVER_HOST}
+  port = 8080
+  port = ${?SERVER_PORT}
+}
+
+database {
+  dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource"
+  dataSource {
+    serverName = "localhost"
+    serverName = ${?DB_HOST}
+    portNumber = 5432
+    portNumber = ${?DB_PORT}
+    databaseName = "summercms"
+    databaseName = ${?DB_NAME}
+    user = "summercms"
+    user = ${?DB_USER}
+    password = "summercms"
+    password = ${?DB_PASSWORD}
+  }
+  connectionTimeout = 30000
+}
+```
+
+### Mill Assembly Configuration
+```scala
+// Source: https://mill-build.org/mill/scalalib/intro.html
+// build.mill (for programmatic config with assembly)
+package build
+import mill.*, scalalib.*, scalalib.assembly.*
+
+object summercms extends ScalaModule:
+  def scalaVersion = "3.8.1"
+  def mvnDeps = Seq(
+    mvn"dev.zio::zio:2.1.14",
+    mvn"dev.zio::zio-http:3.8.1",
+    // ... other deps
+  )
+
+  // Fat JAR configuration
+  def assemblyRules = Seq(
+    Rule.Relocate("META-INF/services/**" -> "META-INF/services/@1"),
+    Rule.ExcludePattern("META-INF/*.SF"),
+    Rule.ExcludePattern("META-INF/*.DSA"),
+    Rule.ExcludePattern("META-INF/*.RSA")
+  )
+```
+
+### Besom AWS Infrastructure
+```scala
+// Source: https://virtuslab.github.io/besom/docs/getting_started/
+// infra/Main.scala (using Scala CLI)
+//> using scala 3.3.1
+//> using dep org.virtuslab::besom-core:0.5.0
+//> using dep org.virtuslab::besom-aws:6.66.0
+
+import besom.*
+import besom.api.aws
+
+@main def main = Pulumi.run {
+  val bucket = aws.s3.Bucket("summercms-assets")
+
+  val db = aws.rds.Instance("summercms-db",
+    aws.rds.InstanceArgs(
+      engine = "postgres",
+      engineVersion = "16.4",
+      instanceClass = "db.t3.micro",
+      allocatedStorage = 20,
+      dbName = "summercms",
+      username = "summercms",
+      password = config.requireSecret("dbPassword"),
+      skipFinalSnapshot = true
+    )
+  )
+
+  Stack.exports(
+    bucketName = bucket.bucket,
+    dbEndpoint = db.endpoint
+  )
+}
+```
+
+## State of the Art
+
+| Old Approach | Current Approach | When Changed | Impact |
+|--------------|------------------|--------------|--------|
+| SBT builds | Mill with YAML config | Mill 1.1.0 (Jan 2026) | 3-7x faster builds, simpler config |
+| Quill Scala 2 | ProtoQuill (Scala 3) | Quill 4.x | Better compile-time metaprogramming |
+| ZIO 1.x service pattern | ZIO 2 ZLayer.make | ZIO 2.0 (2022) | Simpler layer composition |
+| Manual Pulumi Java | Besom Scala SDK | Besom 0.1.0 (2024) | Native Scala 3 IaC |
+| Thread-local transactions | ZIO effect-based transactions | quill-jdbc-zio | No thread-local pollution |
+
+**Deprecated/outdated:**
+- `io.d11` organization for ZIO HTTP - now under `dev.zio`
+- Quill `Connection` dependency - now uses `DataSource`
+- `ZIO.effect` - renamed to `ZIO.attempt` in ZIO 2
+- `ZManaged` - replaced by `ZIO.scoped` in ZIO 2
+
+## Open Questions
+
+Things that couldn't be fully resolved:
+
+1. **Besom + Mill Integration**
+   - What we know: Besom officially only supports Scala CLI, SBT, Maven, Gradle
+   - What's unclear: Whether Mill could work with custom configuration
+   - Recommendation: Use Scala CLI for `infra/` directory (separate from main Mill build)
+
+2. **Hot-Reload in Development**
+   - What we know: Mill has `-w` watch mode, but that recompiles and restarts
+   - What's unclear: True hot-reload without restart for Scala/ZIO
+   - Recommendation: Use Mill `-w` for now; investigate JRebel or ZIO Test live reload for future
+
+3. **Quill Scala 3 Maturity**
+   - What we know: ProtoQuill exists for Scala 3, supports ZIO contexts
+   - What's unclear: Feature parity with Scala 2 Quill, edge case stability
+   - Recommendation: Use quill-jdbc-zio 4.8.6 which supports Scala 3, monitor issues
+
+4. **Transaction Isolation Levels**
+   - What we know: Quill wraps transactions, PostgreSQL supports isolation levels
+   - What's unclear: How to set isolation level per-transaction in quill-jdbc-zio
+   - Recommendation: Use default (READ COMMITTED), document if customization needed
+
+## Sources
+
+### Primary (HIGH confidence)
+- [Mill Build Tool Documentation](https://mill-build.org/mill/index.html) - Build configuration, assembly, watch mode
+- [ZIO HTTP Documentation](https://ziohttp.com/) - Server setup, routes, configuration
+- [ZIO Quill Documentation](https://zio.dev/zio-quill/) - Query DSL, contexts, transactions
+- [Besom Documentation](https://virtuslab.github.io/besom/) - Pulumi Scala SDK setup
+- [ZIO Error Management](https://zio.dev/reference/error-management/best-practices/) - Typed errors, refineOrDie
+
+### Secondary (MEDIUM confidence)
+- [SoftwareMill ZIO 2 Structure](https://softwaremill.com/structuring-zio-2-applications/) - Service layer patterns
+- [ZIO HTTP GitHub Releases](https://github.com/zio/zio-http/releases) - Version 3.8.1 verified
+- [Mill Native Image Plugin](https://github.com/alexarchambault/mill-native-image) - GraalVM integration
+
+### Tertiary (LOW confidence)
+- [Wix ZIO Pitfalls](https://medium.com/wix-engineering/5-pitfalls-to-avoid-when-starting-to-work-with-zio-adefdc7d2d5c) - Common mistakes (older article, patterns still valid)
+- [zio-flyway-db-migrator](https://github.com/DenisNovac/zio-flyway-db-migrator) - Migration pattern example
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH - All libraries verified via official docs and release pages
+- Architecture: HIGH - Patterns from official ZIO documentation and SoftwareMill
+- Pitfalls: MEDIUM - Mix of official docs and community experience posts
+
+**Research date:** 2026-02-04
+**Valid until:** 2026-03-04 (30 days - stable ecosystem)