redpanda using docker-compose

docker-compose/transform/README.adoc

@@ -0,0 +1,73 @@
+= Modify the Wasm Transform in the Quickstart
+
+This directory contains the Go source code (`transform.go`) for the data transform that is used in the Redpanda Self-Managed quickstart.
+If you're following the quickstart, you *do not* need to modify or rebuild this code. The Docker Compose configuration automatically deploys a pre-built transform called `regex.wasm`.
+
+However, if you want to customize the data transform logic, continue reading.
+
+== Why customize the transform?
+
+- **Custom filtering**: Filter by a different regex or apply multiple conditions.
+- **Data manipulation**: Transform records before writing them out. For example, redacting sensitive data or combining fields.
+- **Extended functionality**: Add advanced logging, error handling, or multi-topic routing.
+
+== Prerequisites
+
+You need the following:
+
+- At least Go 1.20 installed.
++
+[source,bash]
+----
+go version
+----
+
+- The Redpanda CLI (`rpk`) installed.
+
+- A running Redpanda cluster. If you're using the local quickstart with Docker Compose, ensure the cluster is up and running. Or, point `rpk` to another Redpanda environment.
+
+== Modify and deploy your transform
+
+. Open link:transform.go[transform.go] and make your changes. For example:
++
+--
+- Change the regex logic to handle different use cases.
+- Add environment variables to control new features.
+- Extend the `doRegexFilter()` function to manipulate records.
+--
+
+. Compile your Go code into a `.wasm` file:
++
+[source,bash]
+----
+rpk transform build
+----
++
+This command compiles your Go source and produces a `.wasm` file that you can deploy to Redpanda.
+
+. Deploy the new transform.
++
+If your Docker Compose setup already has a service to deploy the transform, you can restart that service.
++
+Otherwise, you can deploy your updated `.wasm` manually using `rpk transform deploy`.
+
+. Produce messages into the input topic. For example:
++
+[source,bash]
+----
+echo '{"key":"alice@university.edu","value":"test message"}' | rpk topic produce logins
+----
+
+. Consume from the output topic. For example:
++
+[source,bash]
+----
+rpk topic consume edu-filtered-domains --num 1
+----
+
+== Suggested reading
+
+- link:https://docs.redpanda.com/current/reference/rpk/[Redpanda `rpk` CLI Reference^].
+- link:https://docs.redpanda.com/current/develop/data-transforms/build/[Develop Data Transforms^].
+- https://golang.org/ref/mod[Go Modules^] for managing dependencies and builds in Go.
+- https://docs.docker.com/compose/[Docker Compose^] for customizing your environment.

docker-compose/transform/go.mod

@@ -0,0 +1,5 @@
+module regex
+
+go 1.20
+
+require github.com/redpanda-data/redpanda/src/transform-sdk/go/transform v1.1.0

docker-compose/transform/go.sum

@@ -0,0 +1,2 @@
+github.com/redpanda-data/redpanda/src/transform-sdk/go/transform v1.1.0 h1:KxgHJZsHsrT3YX7DMpu/vJN4TZN3KFm1jzrCFLyOepA=
+github.com/redpanda-data/redpanda/src/transform-sdk/go/transform v1.1.0/go.mod h1:QGgiwwf/BIsD1b7EiyQ/Apzw+RLSpasRDdpOCiefQFQ=

docker-compose/transform/transform.go

@@ -0,0 +1,122 @@
+package main
+// This data transform filters records based on a customizable regex pattern.
+// If a record's key or value
+// (determined by an environment variable) matches the specified regex,
+// the record is forwarded to the output.
+// Otherwise, it is dropped.
+//
+// Usage:
+// 1. Provide the following environment variables in your Docker or configuration setup:
+//    - PATTERN     : (required) a regular expression that determines what you want to match.
+//    - MATCH_VALUE : (optional) a boolean to decide whether to check the record value. If false,
+//                   the record key is checked. Default is false.
+//
+// Example environment variables:
+//   PATTERN=".*\\.edu$"
+//   MATCH_VALUE="true"
+//
+// Logs:
+// This transform logs information about each record and whether it matched.
+// The logs appear in the _redpanda.transform_logs topic, so you can debug how your records are being processed.
+//
+// Build instructions:
+//   go mod tidy
+//   rpk transform build
+//
+// For more details on building transforms with the Redpanda SDK, see:
+// https://docs.redpanda.com/current/develop/data-transforms
+//
+
+import (
+	"log"
+	"os"
+	"regexp"
+	"strings"
+
+	"github.com/redpanda-data/redpanda/src/transform-sdk/go/transform"
+)
+
+var (
+	re         *regexp.Regexp
+	checkValue bool
+)
+
+func isTrueVar(v string) bool {
+	switch strings.ToLower(v) {
+	case "yes", "ok", "1", "true":
+		return true
+	default:
+		return false
+	}
+}
+
+// The main() function runs only once at startup. It performs all initialization steps:
+// - Reads and compiles the regex pattern.
+// - Determines whether to match on the key or value.
+// - Registers the doRegexFilter() function to process records.
+func main() {
+	// Set logging preferences, including timestamp and UTC time.
+	log.SetPrefix("[regex-transform] ")
+	log.SetFlags(log.Ldate | log.Ltime | log.LUTC | log.Lmicroseconds)
+
+	// Start logging the transformation process
+	log.Println("Starting transform...")
+
+	// Read the PATTERN environment variable to get the regex pattern.
+	pattern, ok := os.LookupEnv("PATTERN")
+	if !ok {
+		log.Fatal("Missing PATTERN environment variable")
+	}
+	// Log the regex pattern being used.
+	log.Printf("Using PATTERN: %q\n", pattern)
+	// Compile the regex pattern for later use.
+	re = regexp.MustCompile(pattern)
+
+	// Read the MATCH_VALUE environment variable to determine whether to check the record's value.
+	mk, ok := os.LookupEnv("MATCH_VALUE")
+	checkValue = ok && isTrueVar(mk)
+	log.Printf("MATCH_VALUE set to: %t\n", checkValue)
+
+	log.Println("Initialization complete, waiting for records...")
+
+	// Listen for records to be written, calling doRegexFilter() for each record.
+	transform.OnRecordWritten(doRegexFilter)
+}
+
+// The doRegexFilter() function executes each time a new record is written.
+// It checks whether the record's key or value (based on MATCH_VALUE) matches the compiled regex.
+// If it matches, the record is forwarded, if not, it's dropped.
+func doRegexFilter(e transform.WriteEvent, w transform.RecordWriter) error {
+	// This stores the data to be checked (either the key or value).
+	var dataToCheck []byte
+
+	// Depending on the MATCH_VALUE environment variable, decide whether to check the record's key or value.
+	if checkValue {
+		// Use the value of the record if MATCH_VALUE is true.
+		dataToCheck = e.Record().Value
+		log.Printf("Checking record value: %s\n", string(dataToCheck))
+	} else {
+		// Use the key of the record if MATCH_VALUE is false.
+		dataToCheck = e.Record().Key
+		log.Printf("Checking record key: %s\n", string(dataToCheck))
+	}
+
+	// If there is no key or value to check, log and skip the record.
+	if dataToCheck == nil {
+		log.Println("Record has no key/value to check, skipping.")
+		return nil
+	}
+
+	// Check if the data matches the regex pattern.
+	pass := re.Match(dataToCheck)
+	if pass {
+		// If the record matches the pattern, log and write the record to the output topic.
+		log.Printf("Record matched pattern, passing through. Key: %s, Value: %s\n", string(e.Record().Key), string(e.Record().Value))
+		return w.Write(e.Record())
+	} else {
+		// If the record does not match the pattern, log and drop the record.
+		log.Printf("Record did not match pattern, dropping. Key: %s, Value: %s\n", string(e.Record().Key), string(e.Record().Value))
+		// Do not write the record if it doesn't match the pattern.
+		return nil
+	}
+}

docker-compose/transform/transform.yaml

@@ -0,0 +1,33 @@
+# Transform metadata used by the rpk transform build command.
+# This metadata file tells rpk:
+# 1) The transform’s display name, which also becomes the base for the .wasm file name.
+# 2) A brief description of what it does.
+# 3) Defaults for environment variables.
+# 4) Input and output topics (if you want to define them here rather than in the deploy command).
+
+# Human-readable name of the transform. rpk transform build uses this for the generated .wasm file.
+name: regex
+
+description: |
+  Filters the input topic to records that only match a regular expression.
+
+  Regular expressions are implemented using Go's regexp library, which uses the syntax of RE2.
+  See the RE2 wiki for allowed syntax: https://github.com/google/re2/wiki/Syntax
+
+  Environment variables:
+  - PATTERN: The regular expression that will match against records (required).
+  - MATCH_VALUE: By default, the regex matches keys, but if set to "true", the regex matches values.
+
+# By default, no input topic is set here. (You can set it in your deploy command if preferred.)
+input-topic: ""
+
+# By default, no output topic is set here. (You can set it in your deploy command if preferred.)
+output-topic: ""
+
+# Indicates the specific TinyGo environment used to compile your transform.
+language: tinygo-no-goroutines
+
+env:
+  # The PATTERN variable must be provided at deploy time.
+  # Example: --var=PATTERN=".*@example.com"
+  PATTERN: '<required>'