Add auto-importer for omnivore comments

Taskfile.yml

@@ -17,3 +17,7 @@ tasks:
     cmds:  
       - hugo --cacheDir /tmp/hugo/cache --minify --baseURL "https://www.byjp.me"
       - npm_config_yes=true npx pagefind@latest
+
+  import:
+    cmds: 
+      - cd ./tools/import/omnivore && go run .

assets/scss/_single.scss

@@ -55,6 +55,19 @@
   &-content {
     margin-top: 30px;
 
+    hr {
+      width: 2.5em;
+    }
+
+    footer {
+      text-align: right;
+    }
+  
+    svg {
+      color: var(--accent);
+      vertical-align:middle;
+      margin-left: 0.2em;
+    }
   }
 
   &-content svg, &-info svg {
@@ -534,13 +547,3 @@ figure {
   font-synthesis: none;
 }
 
-.post-content {
-  footer {
-    text-align: right;
-  }
-
-  svg {
-    color: var(--accent);
-    vertical-align:middle;
-  }
-}

content/bookmarks/cultivating-new-ideas.md

@@ -0,0 +1,32 @@
+---
+title: Cultivating new ideas
+date: "2023-11-22T10:00:16Z"
+publishDate: "2023-07-26T14:35:46Z"
+bookmarkOf: https://www.henrikkarlsson.xyz/p/good-ideas
+references:
+  bookmark:
+    url: https://www.henrikkarlsson.xyz/p/good-ideas
+    type: entry
+    name: Cultivating a state of mind where new ideas are born
+    summary: Solitude, creativity, Bergman, Grothendieck, and the pursuit of great
+      ideas.
+    author: Henrik Karlsson, Johanna Wiberg
+---
+
+
+I really enjoyed skimming this article! It’s too long for my distracted brain, but I’ve often thought of “becoming bored” as a part of my creative process. I think that’s part of isolating myself from expectations in much the same way as called out here.
+
+This has also put me to thinking about raising kids; how to ensure there’s always *some* time without peer/social pressure (ie. the absence of consumption devices, like phones with apps)
+
+### Highlights
+
+> He has surrounded himself with people whose influence is the inverse of the social pressure of normal society
+
+---
+
+> “One way to do that is to ask what would be good ideas for _someone else_ to explore. Then your subconscious won't shoot them down to protect you.”
+
+---
+
+> The songs he’s looking for are the ones that he’s ashamed of liking.
+

content/bookmarks/google-as-the-commoditiser-of-the-early-web.md

@@ -0,0 +1,30 @@
+---
+title: Google as the commoditiser of the early web
+date: "2024-01-16T20:50:07Z"
+publishDate: "2023-07-20T00:00:00Z"
+bookmarkOf: https://staltz.com/google-shattered-human-connection.html
+references:
+  bookmark:
+    url: https://staltz.com/google-shattered-human-connection.html
+    type: entry
+    name: Google shattered human connection
+    summary: Open Source Freelancer
+---
+
+
+An interesting take on why the internet can feel soulless, and some implied ways to counter that.
+
+Slightly strong on the “back in the good old days” vibes, but a valid critique of search engines as a remover of mystery. Taking things for granted (“the advert to any question is at my finger tips”) has definitely removed the humanness of information for me.
+
+### Highlights
+
+> **Google popularized the habit of taking things out of context**
+
+---
+
+> Google eliminated the need to connect with communities online if all you wanted was the knowledge produced by that community.
+
+---
+
+> We now assume it is an established truth that the internet is made of “information” or “content”. This has not always been the case.
+

content/bookmarks/google-as-the-commoditiser-of.md

@@ -1,25 +0,0 @@
----
-date: 2023-11-10T09:44:35.051Z
-publishDate: 2024-01-16T20:50:35.051Z
-title: Google as the commoditiser of the early web
-bookmarkOf: https://staltz.com/google-shattered-human-connection.html
-visibility: public
-references:
-  https://staltzCom/googleShatteredHumanConnectionHtml:
-    url: https://staltz.com/google-shattered-human-connection.html
-    type: entry
-    name: André Staltz - Google shattered human connection
-    summary: Open Source Freelancer
----
-
-A really interesting take on why the internet can feel soulless, and some implied ways to counter that.
-
-Highlights:
-
-> Google popularized the habit of taking things out of context.
-
-> Google eliminated the need to connect with communities online if all you wanted was the knowledge produced by that community.
-
-> We now assume it is an established truth that the internet is made of “information” or “content”. This has not always been the case.
-
-

content/bookmarks/the-looking-glass-the-year-of-everyday-risks.md

@@ -0,0 +1,40 @@
+---
+title: 'The Looking Glass: The Year of Everyday Risks'
+date: "2024-01-20T07:57:15Z"
+publishDate: "2024-01-17T16:15:36Z"
+bookmarkOf: https://joulee.medium.com/the-looking-glass-the-year-of-everyday-risks-c46a9f515d3b
+references:
+  bookmark:
+    url: https://joulee.medium.com/the-looking-glass-the-year-of-everyday-risks-c46a9f515d3b
+    type: entry
+    name: 'The Looking Glass: The Year of Everyday Risks'
+    summary: I love their mystery, that tantalizing promise of dreams — what will
+      the year bring? What hopes and hurts, what majesty and mayhem, what lessons
+      and laments? January holds in the palm of its hands…
+    author: Julie Zhuo
+---
+
+I enjoyed this article’s passion for _being alive_, I think there’s a lot of similarity to a previous article’s [obviousness](/tags/obviousness), and putting ourselves in emotionally challenging positions sometimes.
+
+I plan on taking some every day risks this year & always.
+
+### Highlights
+
+> _Aliveness_ is a quality of being. Its opposite is _dullness_.
+
+---
+
+> Aliveness is not just joy, but also its polar opposite, sadness. Aliveness is accepting the risk of both.
+
+---
+
+> Everyday risks are not about the action in of itself; they are about the _feeling_.
+
+---
+
+> Everyday risks — the small actions that come with a twinge of discomfort.
+
+---
+
+> Aliveness is risk, not comfort.
+

content/bookmarks/the-tyrany-of-obviousness.md

@@ -1,20 +1,29 @@
 ---
-date: 2023-12-27T10:18:18.314Z
-publishDate: 2024-01-16T22:10:18.314Z
 title: The Tyrany of Obviousness
+date: "2024-01-20T10:58:27Z"
+publishDate: "2023-12-24T20:51:49Z"
 bookmarkOf: https://mentalhellth.xyz/p/breaking-the-tyranny-of-obviousness
 references:
-  https://mentalhellthXyz/p/breakingTheTyrannyOfObviousness:
+  bookmark:
     url: https://mentalhellth.xyz/p/breaking-the-tyranny-of-obviousness
     type: entry
     name: Breaking the Tyranny of Obviousness
     summary: We are stuck in a hell of frictionlessness.
-    featured: https://substackcdn.com/image/fetch/w_1200,h_600,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175bc33-c1e8-4b34-a644-217588d1adcd_937x489.png
     author: P.E. Moskowitz
 ---
 
-A superb read, especially for me as I look for more depth in life. I keep finding myself coming back to this thought:
 
-> There are many forces preventing us from feeling abstractly and confusingly and deeply, and from producing things that help others feel so.
+A superb read, especially for me as I look for more depth in life.
+
+I think I've certainly trained myself for a frictionless existence, and have (up until the last few years) tended to avoid difficult thoughts — but I feel so much richer for embracing them!
+
+### Highlights
+
+> “A picture lives by companionship, expanding and quickening in the eyes of the sensitive observer,” [Rothko once wrote](https://www.nytimes.com/1970/02/26/archives/mark-rothko-artist-a-suicide-here-at-66-mark-rothko-abstract.html). “It dies by the same token. It is therefore a risky and unfeeling act to send it out into the world. How often it must be permanently impaired by the eyes of the vulgar and the cruelty of the impotent who would extend the affliction universally!”
+
+---
+
+> there are many forces preventing us from feeling abstractly and confusingly and deeply, and from producing things that help others feel so.
+
+I keep finding myself coming back to this thought in particular. I _want_ to feel abstractly and deeply — it feels much more like "living".
 
-I think I've certainly trained myself for a frictionless life, and have (up until the last few years) tended to avoid difficult thoughts — but I feel so much richer for embracing them!

layouts/_default/single.html

@@ -22,12 +22,19 @@
           {{ partial "year-relative-date.html" $date }}
         </p>
       {{ end }}
+      <p>
+        <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-compass">
+          <circle cx="12" cy="12" r="10"></circle>
+          <polygon points="16.24 7.76 14.12 14.12 7.76 16.24 9.88 9.88 16.24 7.76"></polygon>
+        </svg>
+        Back to <a href="{{ .Page.Parent.RelPermalink }}">{{ .Page.Parent.Title }}</a>
+      </p>
       <p class="reading-time">
         <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clock">
           <circle cx="12" cy="12" r="10"></circle>
           <polyline points="12 6 12 12 16 14"></polyline>
         </svg>
-        {{ i18n "readingTime" .Page.ReadingTime }}
+        {{ i18n "readingTime" .Page.ReadingTime }} read
       </p>
       <p class="author h-card hidden" aria-hidden="true">
         <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-user"><path d="M20 21v-2a4 4 0 0 0-4-4H8a4 4 0 0 0-4 4v2"></path><circle cx="12" cy="7" r="4"></circle></svg>

tools/import/omnivore/.gitignore

@@ -0,0 +1 @@
+.env

tools/import/omnivore/go.mod

@@ -0,0 +1,5 @@
+module github.com/by-jp/www.byjp.me/tools/import/omnivore
+
+go 1.21.6
+
+require gopkg.in/yaml.v2 v2.4.0

tools/import/omnivore/go.sum

@@ -0,0 +1,2 @@
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

tools/import/omnivore/go.work

@@ -0,0 +1,3 @@
+go 1.21.6
+
+use .

tools/import/omnivore/go.work.sum

@@ -0,0 +1 @@
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

tools/import/omnivore/main.go

@@ -0,0 +1,325 @@
+package main
+
+import (
+	"bytes"
+	_ "embed"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path"
+	"regexp"
+	"strings"
+	"time"
+
+	"gopkg.in/yaml.v2"
+)
+
+//go:embed query.gql
+var gql string
+
+func main() {
+	apiKey, ok := os.LookupEnv("OMNIVORE_API_KEY")
+	if !ok || len(apiKey) == 0 {
+		fmt.Fprint(os.Stderr, "OMNIVORE_API_KEY is not set")
+		os.Exit(1)
+	}
+	// Make the GraphQL request
+	articles, err := omnivoreArticles(
+		"in:archive has:highlights is:read sort:updated-des",
+		apiKey,
+	)
+	if err != nil {
+		fmt.Println("Failed retrieve articles:", err)
+		os.Exit(1)
+	}
+
+	outputDir := "../../../content/bookmarks"
+
+	for _, article := range articles {
+		if err := outputArticle(article, outputDir); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to output article: %v\n", err)
+		}
+	}
+}
+
+var hashtags = regexp.MustCompile(`#\w+`)
+
+func outputArticle(article Article, outputDir string) error {
+
+	slug := kebab(article.Title)
+	hugoPost, err := os.Create(path.Join(outputDir, fmt.Sprintf("%s.md", slug)))
+	if err != nil {
+		return err
+	}
+
+	fm := FrontMatter{
+		Title:       article.Title,
+		Date:        article.BookmarkDate.Format(time.RFC3339),
+		PublishDate: article.PublishDate.Format(time.RFC3339),
+		BookmarkOf:  article.OriginalURL,
+		References: map[string]Ref{
+			"bookmark": {
+				URL:     article.OriginalURL,
+				Type:    "entry",
+				Name:    article.OriginalTitle,
+				Summary: article.OriginalSummary,
+				Author:  article.OriginalAuthor,
+			},
+		},
+	}
+
+	fmt.Fprintln(hugoPost, "---")
+
+	if err := yaml.NewEncoder(hugoPost).Encode(fm); err != nil {
+		return err
+	}
+
+	fmt.Fprint(hugoPost, "---\n\n")
+	fmt.Fprintln(hugoPost, linkHashtags(article.Annonation))
+	fmt.Fprintln(hugoPost)
+	fmt.Fprint(hugoPost, "### Highlights\n\n")
+	for i, highlight := range article.Highlights {
+		noTrailingNewLine := strings.TrimRight(highlight.Quote, "\n ")
+		quote := "> " + strings.ReplaceAll(noTrailingNewLine, "\n", "\n> ")
+		fmt.Fprint(hugoPost, quote+"\n\n")
+
+		if highlight.Comment != "" {
+			fmt.Fprint(hugoPost, linkHashtags(highlight.Comment)+"\n\n")
+		}
+
+		if i < len(article.Highlights)-1 {
+			fmt.Fprint(hugoPost, "---\n\n")
+		}
+	}
+
+	return nil
+}
+
+func linkHashtags(text string) string {
+	return hashtags.ReplaceAllStringFunc(text, func(hashtag string) string {
+		return fmt.Sprintf("[%s](/tags/%s)", hashtag[1:], hashtag[1:])
+	})
+}
+
+var kebaber = regexp.MustCompile(`[^a-zA-Z0-9]+`)
+
+func kebab(str string) string {
+	return kebaber.ReplaceAllString(strings.ToLower(str), "-")
+}
+
+type GraphQLRequest struct {
+	Query     string                 `json:"query"`
+	Variables map[string]interface{} `json:"variables"`
+}
+
+const omnivoreEndpoint = "https://api-prod.omnivore.app/api/graphql"
+
+type Article struct {
+	ID              string
+	Title           string
+	BookmarkDate    time.Time
+	PublishDate     time.Time
+	OriginalTitle   string
+	OriginalURL     string
+	OriginalSummary string
+	OriginalAuthor  string
+	Annonation      string
+	Highlights      []ArticleHighlight
+}
+
+type ArticleHighlight struct {
+	Quote   string
+	Comment string
+}
+
+type FrontMatter struct {
+	Title       string
+	Date        string
+	PublishDate string `yaml:"publishDate"`
+	BookmarkOf  string `yaml:"bookmarkOf"`
+	References  map[string]Ref
+}
+
+type Ref struct {
+	URL     string `yaml:"url"`
+	Type    string `yaml:"type"`
+	Name    string `yaml:"name"`
+	Summary string `yaml:"summary,omitempty"`
+	Author  string `yaml:"author,omitempty"`
+}
+
+type SearchResults struct {
+	Data struct {
+		Search struct {
+			Edges []struct {
+				Node SearchResult
+			}
+			PageInfo struct {
+				HasNextPage bool   `json:"hasNextPage"`
+				EndCursor   string `json:"endCursor"`
+			}
+		}
+	}
+}
+
+type SearchResult struct {
+	ID                 string `json:"id"`
+	Title              string `json:"title"`
+	OriginalArticleURL string `json:"originalArticleUrl"`
+	Author             string `json:"author"`
+	PublishedAt        string `json:"publishedAt"`
+	ReadAt             string `json:"readAt"`
+	Description        string `json:"description"`
+	Highlights         []Highlight
+	Labels             []struct {
+		Name string `json:"name"`
+	}
+}
+
+type Highlight struct {
+	Type       string  `json:"type"`
+	Position   float64 `json:"highlightPositionPercent"`
+	Annotation string  `json:"annotation"`
+	Quote      string  `json:"quote"`
+}
+
+func omnivoreArticles(query string, apiKey string) ([]Article, error) {
+	cursor := ""
+	var articles []Article
+	for {
+		newArticles, nextCursor, err := omnivoreRequest(query, apiKey, cursor)
+		if err != nil {
+			return nil, err
+		}
+
+		articles = append(articles, newArticles...)
+		if len(nextCursor) == 0 {
+			break
+		}
+		cursor = nextCursor
+	}
+	return articles, nil
+}
+
+func omnivoreRequest(query, apiKey, cursor string) ([]Article, string, error) {
+	request := GraphQLRequest{
+		Query: gql,
+		Variables: map[string]interface{}{
+			"query": query,
+			"after": cursor,
+		},
+	}
+
+	requestJSON, err := json.Marshal(request)
+	if err != nil {
+		return nil, "", err
+	}
+
+	req, err := http.NewRequest("POST", omnivoreEndpoint, bytes.NewBuffer(requestJSON))
+	if err != nil {
+		return nil, "", err
+	}
+
+	req.Header.Set("Authorization", apiKey)
+	req.Header.Set("Content-Type", "application/json")
+
+	client := &http.Client{}
+	res, err := client.Do(req)
+	if err != nil {
+		return nil, "", err
+	}
+	defer res.Body.Close()
+
+	if res.StatusCode != http.StatusOK {
+		return nil, "", fmt.Errorf("unexpected status code: %d", res.StatusCode)
+	}
+
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return nil, "", err
+	}
+
+	return parseResponse(body)
+}
+
+var titleSplitter = regexp.MustCompile(`\s[-–—|]\s`)
+
+func parseResponse(body []byte) ([]Article, string, error) {
+	var searchResults SearchResults
+	if err := json.Unmarshal(body, &searchResults); err != nil {
+		return nil, "", err
+	}
+
+	var articles []Article
+	for _, edge := range searchResults.Data.Search.Edges {
+		sr := edge.Node
+
+		var highlights []ArticleHighlight
+		var annotation string
+		for _, highlight := range sr.Highlights {
+			if highlight.Type == "NOTE" {
+				annotation = highlight.Annotation
+			} else {
+				highlights = append(highlights, ArticleHighlight{
+					Quote:   highlight.Quote,
+					Comment: highlight.Annotation,
+				})
+			}
+		}
+
+		if len(annotation) == 0 {
+			continue
+		}
+
+		bookmarked, err := time.Parse(time.RFC3339, sr.ReadAt)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to parse ReadAt date: %s\n", sr.ID)
+			continue
+		}
+		published, err := time.Parse(time.RFC3339, sr.PublishedAt)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to parse PublishedAt date: %s\n", sr.ID)
+			continue
+		}
+
+		abbreviatedOriginalTitle := sr.Title
+		if parts := titleSplitter.Split(abbreviatedOriginalTitle, -1); len(parts) > 1 {
+			abbreviatedOriginalTitle = parts[0]
+			for _, part := range parts[1:] {
+				if len(part) > len(abbreviatedOriginalTitle) {
+					abbreviatedOriginalTitle = part
+				}
+			}
+		}
+		title := abbreviatedOriginalTitle
+		if annotation[0:2] == "# " {
+			parts := strings.SplitN(annotation, "\n", 2)
+			title = parts[0][2:]
+			annotation = parts[1]
+		}
+
+		article := Article{
+			ID:              sr.ID,
+			Title:           title,
+			OriginalTitle:   abbreviatedOriginalTitle,
+			OriginalURL:     sr.OriginalArticleURL,
+			OriginalAuthor:  sr.Author,
+			OriginalSummary: sr.Description,
+			BookmarkDate:    bookmarked,
+			PublishDate:     published,
+			Highlights:      highlights,
+			Annonation:      annotation,
+		}
+
+		articles = append(articles, article)
+	}
+
+	var cursor string
+	if searchResults.Data.Search.PageInfo.HasNextPage {
+		cursor = searchResults.Data.Search.PageInfo.EndCursor
+	}
+
+	return articles, cursor, nil
+}

tools/import/omnivore/query.gql

@@ -0,0 +1,46 @@
+query Search(
+  $after: String
+  $first: Int
+  $query: String
+  $includeContent: Boolean
+  $format: String
+) {
+  search(
+    first: $first,
+    after: $after,
+    query: $query,
+    includeContent: $includeContent,
+    format: $format
+  ) {
+    ... on SearchSuccess {
+      edges {
+        node {
+          id
+          title
+          originalArticleUrl
+          author
+          publishedAt
+          readAt
+          description
+          highlights {
+            type
+            highlightPositionPercent
+            annotation
+            quote
+          }
+          labels {
+            name
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+        totalCount
+      }
+    }
+    ... on SearchError {
+      errorCodes
+    }
+  }
+}