summercms-initial-research/.planning/phases/01-foundation/01-RESEARCH.md

# Phase 1: Foundation - Research

**Researched:** 2026-02-04
**Domain:** Scala/ZIO stack with Mill build, PostgreSQL/Quill, and Pulumi deployment
**Confidence:** HIGH

## Summary

This phase establishes the foundational Scala/ZIO stack for SummerCMS. The research validates a modern, well-supported stack: Mill 1.1.x as build tool, ZIO HTTP 3.8.x for the HTTP server, Quill 4.8.x with quill-jdbc-zio for compile-time SQL validation against PostgreSQL, Flyway for database migrations (wrapped in ZIO effects), and Besom 0.5.x for Pulumi-based infrastructure as code.

The stack is coherent and production-ready. Mill provides fast builds with YAML-based declarative configuration and fat JAR assembly. ZIO HTTP offers high-performance Netty-backed HTTP with native ZIO integration. Quill delivers compile-time SQL validation with PostgreSQL support. Flyway is the standard JVM migration tool with established ZIO wrappers. Besom brings Pulumi to Scala 3 with pure functional semantics.

**Primary recommendation:** Use Mill 1.1.x with declarative `build.mill.yaml` for simplicity, ZIO HTTP 3.8.x with zio-config-typesafe for HOCON configuration, Quill 4.8.6 via quill-jdbc-zio for database access, Flyway wrapped in ZIO effects for migrations, and Besom 0.5.x with Scala CLI for Pulumi deployment.

## Standard Stack

The established libraries/tools for this domain:

### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| Mill | 1.1.1 | Build tool | 3-7x faster than Maven/Gradle, declarative YAML config, native Scala support |
| ZIO | 2.1.x | Effect system | Industry-standard FP effect library for Scala, excellent ecosystem |
| ZIO HTTP | 3.8.1 | HTTP server | Netty-backed, native ZIO, high concurrency via fibers |
| Quill | 4.8.6 | Database queries | Compile-time SQL validation, type-safe, PostgreSQL support |
| Flyway | 10.x | Migrations | JVM standard, version tracking, rollback support |
| Besom | 0.5.0 | IaC (Pulumi) | Scala 3 native Pulumi SDK, functional semantics |

### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| zio-config | 4.0.5 | Configuration | HOCON file loading with environment overrides |
| zio-config-typesafe | 4.0.5 | HOCON support | Parse application.conf files |
| zio-config-magnolia | 4.0.2 | Config derivation | Auto-derive config from case classes |
| postgresql | 42.7.x | JDBC driver | PostgreSQL database connectivity |
| HikariCP | 5.x | Connection pooling | Built into quill-jdbc-zio |

### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Mill | SBT | SBT has larger ecosystem but slower, more complex |
| Mill | Scala CLI | Scala CLI simpler for small projects, Mill better for larger ones |
| Quill | Doobie | Doobie uses SQL strings, Quill has compile-time validation |
| Flyway | Liquibase | Liquibase XML-heavy, Flyway simpler for SQL migrations |
| Besom | Pulumi Java SDK | Java SDK exists but Besom is Scala-native, better FP support |

**Installation (Mill build.mill.yaml):**
```yaml
extends: ScalaModule
scalaVersion: 3.8.1
mvnDeps:
  - dev.zio::zio:2.1.14
  - dev.zio::zio-http:3.8.1
  - dev.zio::zio-config:4.0.5
  - dev.zio::zio-config-typesafe:4.0.5
  - dev.zio::zio-config-magnolia:4.0.2
  - io.getquill::quill-jdbc-zio:4.8.6
  - org.postgresql:postgresql:42.7.4
  - org.flywaydb:flyway-core:10.23.0
  - org.flywaydb:flyway-database-postgresql:10.23.0
```

## Architecture Patterns

### Recommended Project Structure
```
summercms/
├── build.mill.yaml           # Mill declarative config
├── src/
│   ├── Main.scala            # Application entry point
│   ├── config/
│   │   └── AppConfig.scala   # Configuration case classes
│   ├── db/
│   │   ├── Migrator.scala    # Flyway wrapper in ZIO
│   │   └── QuillContext.scala # Quill PostgreSQL context
│   ├── repository/
│   │   ├── Repository.scala  # Base repository trait
│   │   └── UserRepository.scala # Example repository
│   ├── service/
│   │   └── UserService.scala # Business logic layer
│   └── api/
│       ├── Routes.scala      # HTTP route definitions
│       └── HealthRoutes.scala # Health check endpoints
├── resources/
│   ├── application.conf      # HOCON configuration
│   └── db/migration/         # Flyway SQL migrations
├── test/
│   └── src/                  # Test sources
└── infra/                    # Pulumi/Besom infrastructure
    ├── project.scala         # Scala CLI project config
    └── Main.scala            # Infrastructure definition
```

### Pattern 1: ZIO Service Layer (Trait + Live)
**What:** Define services as traits with ZLayer-based implementations
**When to use:** All service components (repositories, business services)
**Example:**
```scala
// Source: https://softwaremill.com/structuring-zio-2-applications/
// Trait defines the contract
trait UserRepository:
  def findById(id: Long): IO[RepositoryError, Option[User]]
  def create(user: User): IO[RepositoryError, User]
  def update(user: User): IO[RepositoryError, User]
  def delete(id: Long): IO[RepositoryError, Unit]

// Companion object provides ZLayer
object UserRepository:
  val live: ZLayer[Quill.Postgres[SnakeCase], Nothing, UserRepository] =
    ZLayer.fromFunction(UserRepositoryLive(_))

// Implementation class
class UserRepositoryLive(quill: Quill.Postgres[SnakeCase]) extends UserRepository:
  import quill.*

  def findById(id: Long): IO[RepositoryError, Option[User]] =
    run(query[User].filter(_.id == lift(id)))
      .map(_.headOption)
      .mapError(e => RepositoryError.DatabaseError(e.getMessage))
```

### Pattern 2: Typed Error ADT
**What:** Define domain errors as sealed traits for exhaustive handling
**When to use:** Repository and service error types
**Example:**
```scala
// Source: https://zio.dev/reference/error-management/best-practices/unexpected-errors/
sealed trait RepositoryError
object RepositoryError:
  case class NotFound(entity: String, id: Long) extends RepositoryError
  case class Conflict(entity: String, message: String) extends RepositoryError
  case class ValidationError(errors: List[String]) extends RepositoryError
  case class DatabaseError(message: String) extends RepositoryError

// Use refineOrDie for unexpected errors
def findById(id: Long): IO[RepositoryError, Option[User]] =
  run(query[User].filter(_.id == lift(id)))
    .map(_.headOption)
    .refineOrDie {
      case e: SQLException => RepositoryError.DatabaseError(e.getMessage)
    }
```

### Pattern 3: HOCON Configuration with ZIO Config
**What:** Type-safe configuration from HOCON files with environment overrides
**When to use:** Application configuration (server, database, etc.)
**Example:**
```scala
// Source: https://ziohttp.com/guides/integration-with-zio-config
// Config case class
case class AppConfig(
  server: ServerConfig,
  database: DatabaseConfig
)
case class ServerConfig(host: String, port: Int)
case class DatabaseConfig(
  host: String,
  port: Int,
  database: String,
  user: String,
  password: String
)

// application.conf
// server {
//   host = "0.0.0.0"
//   host = ${?SERVER_HOST}
//   port = 8080
//   port = ${?SERVER_PORT}
// }

// Bootstrap with config provider
object Main extends ZIOAppDefault:
  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.setConfigProvider(ConfigProvider.fromResourcePath())
```

### Pattern 4: Quill PostgreSQL Context with ZIO
**What:** Configure Quill for compile-time validated PostgreSQL queries
**When to use:** All database access
**Example:**
```scala
// Source: https://zio.dev/zio-quill/getting-started/
import io.getquill.*
import io.getquill.jdbczio.Quill

// Create the Quill layer
val quillLayer: ZLayer[DataSource, Nothing, Quill.Postgres[SnakeCase]] =
  Quill.Postgres.fromNamingStrategy(SnakeCase)

// DataSource from config
val dataSourceLayer: ZLayer[Any, Throwable, DataSource] =
  Quill.DataSource.fromPrefix("database")

// Combine layers
val dbLayer = dataSourceLayer >>> quillLayer
```

### Pattern 5: Flyway Migrations Wrapped in ZIO
**What:** Run database migrations as ZIO effects
**When to use:** Application startup, CLI commands
**Example:**
```scala
// Source: https://github.com/DenisNovac/zio-flyway-db-migrator (pattern)
import org.flywaydb.core.Flyway
import zio.*

trait Migrator:
  def migrate: Task[Int]
  def rollback: Task[Unit]
  def status: Task[MigrationStatus]

object Migrator:
  val live: ZLayer[DataSource, Nothing, Migrator] =
    ZLayer.fromFunction { (ds: DataSource) =>
      new Migrator:
        private val flyway = Flyway.configure()
          .dataSource(ds)
          .locations("classpath:db/migration")
          .table("summer_migrations")
          .load()

        def migrate: Task[Int] = ZIO.attempt(flyway.migrate().migrationsExecuted)
        def rollback: Task[Unit] = ZIO.attempt(flyway.undo()).unit
        def status: Task[MigrationStatus] = ZIO.attempt {
          val info = flyway.info()
          MigrationStatus(
            current = Option(info.current()).map(_.getVersion.toString),
            pending = info.pending().length
          )
        }
    }
```

### Anti-Patterns to Avoid
- **Using `ZIO.effect` for Futures:** Use `ZIO.fromFuture` instead to properly await async completion
- **Discarding effects in for-comprehensions:** Always chain effects with `flatMap` or `*>`
- **Double-wrapping ZIO in pattern matches:** Evaluate each case branch separately
- **Using `println` instead of ZIO Console:** Use `Console.printLine` for proper effect tracking
- **Returning ZIO inside yield:** The yield should return values, not effects
- **Typing unexpected errors:** Use `refineOrDie` to only type recoverable errors

## Don't Hand-Roll

Problems that look simple but have existing solutions:

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| SQL query building | String concatenation | Quill QDSL | SQL injection, type safety, compile-time validation |
| Database migrations | Custom version tracking | Flyway | Checksums, rollbacks, history table, team coordination |
| Configuration loading | Manual parsing | zio-config-typesafe | Environment overrides, nested configs, validation |
| Connection pooling | Manual pool management | HikariCP (via Quill) | Connection lifecycle, leak detection, metrics |
| HTTP routing | Manual path matching | ZIO HTTP Routes | Type-safe path params, method matching, middleware |
| Error handling | Try/catch | ZIO typed errors | Composability, resource safety, stack traces |
| Dependency injection | Manual wiring | ZLayer | Compile-time verification, resource management |
| Fat JAR building | Manual manifest/classpath | Mill assembly | Conflict resolution, main class detection |
| Infrastructure code | CloudFormation/Terraform | Besom (Pulumi) | Type safety, Scala ecosystem, refactoring support |

**Key insight:** The ZIO ecosystem provides cohesive solutions that work together. Hand-rolling any component breaks the effect composition model and loses compile-time guarantees.

## Common Pitfalls

### Pitfall 1: Quill Variable Name Conflict
**What goes wrong:** Naming a variable `io` causes compiler errors with Quill
**Why it happens:** Quill's package is `io.getquill`, causing naming collision
**How to avoid:** Never use `io` as a variable name in files importing Quill
**Warning signs:** Strange "not found: value io" errors after adding Quill

### Pitfall 2: Wrong ZIO Effect Wrapping for Futures
**What goes wrong:** Async operations complete prematurely or run on wrong thread pool
**Why it happens:** Using `ZIO.attempt` instead of `ZIO.fromFuture` for Future-returning methods
**How to avoid:** Always use `ZIO.fromFuture` for methods returning `Future[A]`
**Warning signs:** Unexpected null values, race conditions, operations not completing

### Pitfall 3: Nested Transaction Failures in Quill
**What goes wrong:** Foreign key constraint violations in nested transactions
**Why it happens:** PostgreSQL async driver starts separate transactions for nested calls
**How to avoid:** Flatten transaction logic, avoid nesting `ctx.transaction` blocks
**Warning signs:** FK constraint errors that work in simple tests but fail in complex flows

### Pitfall 4: Missing Effect Chaining in For-Comprehensions
**What goes wrong:** Effects are silently discarded, side effects don't execute
**Why it happens:** Not using `<-` or `*>` to chain effects, just sequencing statements
**How to avoid:** Use `_ <- effect` or `effect1 *> effect2` for side-effect-only operations
**Warning signs:** Logging statements not appearing, database writes not persisting

### Pitfall 5: Besom Resource Laziness
**What goes wrong:** Cloud resources not deployed despite being defined
**Why it happens:** Besom resources are lazy and must be referenced to deploy
**How to avoid:** Always export or reference resources in Stack.exports or other resources
**Warning signs:** `pulumi up` shows no changes when resources are defined

### Pitfall 6: Mill Not Supported by Besom
**What goes wrong:** Cannot use Mill for Pulumi infrastructure projects
**Why it happens:** Besom only supports Scala CLI, SBT, Maven, Gradle (not Mill)
**How to avoid:** Use Scala CLI for the `infra/` directory, Mill for main application
**Warning signs:** Besom compilation failures when using Mill

### Pitfall 7: GraalVM Native Image Reflection Issues
**What goes wrong:** Runtime errors in native image builds
**Why it happens:** Native image requires reflection configuration for frameworks
**How to avoid:** Start with fat JAR, only optimize to native image when needed
**Warning signs:** ClassNotFoundException, NoSuchMethodError at runtime in native builds

## Code Examples

Verified patterns from official sources:

### Basic ZIO HTTP Server
```scala
// Source: https://ziohttp.com/
import zio.*
import zio.http.*

object Main extends ZIOAppDefault:
  val routes = Routes(
    Method.GET / "health" -> handler(Response.text("ok")),
    Method.GET / "ready" -> handler {
      // Check database connectivity
      ZIO.serviceWithZIO[DataSource](ds =>
        ZIO.attempt(ds.getConnection.close())
      ).as(Response.text("ready"))
       .catchAll(_ => ZIO.succeed(Response.status(Status.ServiceUnavailable)))
    }
  )

  def run = Server.serve(routes).provide(
    Server.defaultWithPort(8080),
    dataSourceLayer
  )
```

### Quill PostgreSQL Query
```scala
// Source: https://zio.dev/zio-quill/getting-started/
import io.getquill.*
import io.getquill.jdbczio.Quill

case class User(id: Long, email: String, createdAt: java.time.Instant)

class UserRepositoryLive(quill: Quill.Postgres[SnakeCase]) extends UserRepository:
  import quill.*

  inline def users = quote(querySchema[User]("summer_users"))

  def findById(id: Long): Task[Option[User]] =
    run(users.filter(_.id == lift(id))).map(_.headOption)

  def create(user: User): Task[User] =
    run(users.insertValue(lift(user)).returning(u => u))

  def findByEmail(email: String): Task[Option[User]] =
    run(users.filter(_.email == lift(email))).map(_.headOption)
```

### HOCON Configuration
```hocon
# Source: https://ziohttp.com/guides/integration-with-zio-config
# resources/application.conf

server {
  host = "0.0.0.0"
  host = ${?SERVER_HOST}
  port = 8080
  port = ${?SERVER_PORT}
}

database {
  dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource"
  dataSource {
    serverName = "localhost"
    serverName = ${?DB_HOST}
    portNumber = 5432
    portNumber = ${?DB_PORT}
    databaseName = "summercms"
    databaseName = ${?DB_NAME}
    user = "summercms"
    user = ${?DB_USER}
    password = "summercms"
    password = ${?DB_PASSWORD}
  }
  connectionTimeout = 30000
}
```

### Mill Assembly Configuration
```scala
// Source: https://mill-build.org/mill/scalalib/intro.html
// build.mill (for programmatic config with assembly)
package build
import mill.*, scalalib.*, scalalib.assembly.*

object summercms extends ScalaModule:
  def scalaVersion = "3.8.1"
  def mvnDeps = Seq(
    mvn"dev.zio::zio:2.1.14",
    mvn"dev.zio::zio-http:3.8.1",
    // ... other deps
  )

  // Fat JAR configuration
  def assemblyRules = Seq(
    Rule.Relocate("META-INF/services/**" -> "META-INF/services/@1"),
    Rule.ExcludePattern("META-INF/*.SF"),
    Rule.ExcludePattern("META-INF/*.DSA"),
    Rule.ExcludePattern("META-INF/*.RSA")
  )
```

### Besom AWS Infrastructure
```scala
// Source: https://virtuslab.github.io/besom/docs/getting_started/
// infra/Main.scala (using Scala CLI)
//> using scala 3.3.1
//> using dep org.virtuslab::besom-core:0.5.0
//> using dep org.virtuslab::besom-aws:6.66.0

import besom.*
import besom.api.aws

@main def main = Pulumi.run {
  val bucket = aws.s3.Bucket("summercms-assets")

  val db = aws.rds.Instance("summercms-db",
    aws.rds.InstanceArgs(
      engine = "postgres",
      engineVersion = "16.4",
      instanceClass = "db.t3.micro",
      allocatedStorage = 20,
      dbName = "summercms",
      username = "summercms",
      password = config.requireSecret("dbPassword"),
      skipFinalSnapshot = true
    )
  )

  Stack.exports(
    bucketName = bucket.bucket,
    dbEndpoint = db.endpoint
  )
}
```

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| SBT builds | Mill with YAML config | Mill 1.1.0 (Jan 2026) | 3-7x faster builds, simpler config |
| Quill Scala 2 | ProtoQuill (Scala 3) | Quill 4.x | Better compile-time metaprogramming |
| ZIO 1.x service pattern | ZIO 2 ZLayer.make | ZIO 2.0 (2022) | Simpler layer composition |
| Manual Pulumi Java | Besom Scala SDK | Besom 0.1.0 (2024) | Native Scala 3 IaC |
| Thread-local transactions | ZIO effect-based transactions | quill-jdbc-zio | No thread-local pollution |

**Deprecated/outdated:**
- `io.d11` organization for ZIO HTTP - now under `dev.zio`
- Quill `Connection` dependency - now uses `DataSource`
- `ZIO.effect` - renamed to `ZIO.attempt` in ZIO 2
- `ZManaged` - replaced by `ZIO.scoped` in ZIO 2

## Open Questions

Things that couldn't be fully resolved:

1. **Besom + Mill Integration**
   - What we know: Besom officially only supports Scala CLI, SBT, Maven, Gradle
   - What's unclear: Whether Mill could work with custom configuration
   - Recommendation: Use Scala CLI for `infra/` directory (separate from main Mill build)

2. **Hot-Reload in Development**
   - What we know: Mill has `-w` watch mode, but that recompiles and restarts
   - What's unclear: True hot-reload without restart for Scala/ZIO
   - Recommendation: Use Mill `-w` for now; investigate JRebel or ZIO Test live reload for future

3. **Quill Scala 3 Maturity**
   - What we know: ProtoQuill exists for Scala 3, supports ZIO contexts
   - What's unclear: Feature parity with Scala 2 Quill, edge case stability
   - Recommendation: Use quill-jdbc-zio 4.8.6 which supports Scala 3, monitor issues

4. **Transaction Isolation Levels**
   - What we know: Quill wraps transactions, PostgreSQL supports isolation levels
   - What's unclear: How to set isolation level per-transaction in quill-jdbc-zio
   - Recommendation: Use default (READ COMMITTED), document if customization needed

## Sources

### Primary (HIGH confidence)
- [Mill Build Tool Documentation](https://mill-build.org/mill/index.html) - Build configuration, assembly, watch mode
- [ZIO HTTP Documentation](https://ziohttp.com/) - Server setup, routes, configuration
- [ZIO Quill Documentation](https://zio.dev/zio-quill/) - Query DSL, contexts, transactions
- [Besom Documentation](https://virtuslab.github.io/besom/) - Pulumi Scala SDK setup
- [ZIO Error Management](https://zio.dev/reference/error-management/best-practices/) - Typed errors, refineOrDie

### Secondary (MEDIUM confidence)
- [SoftwareMill ZIO 2 Structure](https://softwaremill.com/structuring-zio-2-applications/) - Service layer patterns
- [ZIO HTTP GitHub Releases](https://github.com/zio/zio-http/releases) - Version 3.8.1 verified
- [Mill Native Image Plugin](https://github.com/alexarchambault/mill-native-image) - GraalVM integration

### Tertiary (LOW confidence)
- [Wix ZIO Pitfalls](https://medium.com/wix-engineering/5-pitfalls-to-avoid-when-starting-to-work-with-zio-adefdc7d2d5c) - Common mistakes (older article, patterns still valid)
- [zio-flyway-db-migrator](https://github.com/DenisNovac/zio-flyway-db-migrator) - Migration pattern example

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH - All libraries verified via official docs and release pages
- Architecture: HIGH - Patterns from official ZIO documentation and SoftwareMill
- Pitfalls: MEDIUM - Mix of official docs and community experience posts

**Research date:** 2026-02-04
**Valid until:** 2026-03-04 (30 days - stable ecosystem)