gist backup

github官方並沒有提供gist的備份方式，不過提供了功能充份的API，目前已經到了v3。
https://developer.github.com/v3/guides/getting-started/
https://developer.github.com/v3/auth/

backup public gists

這裡使用v1版的API即可，v1當時也沒有功能能備份private gists。
但也就是說你只要知道github帳號就可以備份任意人的public gists。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/usr/bin/env python
# @see https://stackoverflow.com/questions/6724490/pull-all-gists-from-github
# Clone or update all a user's gists
# curl -ks https://gist.githubusercontent.com/fedir/5466075/raw/gist-backup.py | USER=fedir python
# USER=fedir python gist-backup.py

import json
import urllib
from subprocess import call
from urllib import urlopen
import os
import math
USER = os.environ['USER']

perpage=30.0
userurl = urlopen('https://api.github.com/users/' + USER)
public_gists = json.load(userurl)
gistcount = public_gists['public_gists']
print "Found gists : " + str(gistcount)
pages = int(math.ceil(float(gistcount)/perpage))
print "Found pages : " + str(pages)

f=open('./contents.txt', 'w+')

for page in range(pages):
	pageNumber = str(page + 1)
	print "Processing page number " + pageNumber
	pageUrl = 'https://api.github.com/users/' + USER  + '/gists?page=' + pageNumber + '&per_page=' + str(int(perpage))
	u = urlopen (pageUrl)
	gists = json.load(u)
	startd = os.getcwd()
	for gist in gists:
		gistd = gist['id']
		gistUrl = 'git://gist.github.com/' + gistd + '.git'	
		if os.path.isdir(gistd):
			os.chdir(gistd)
			call(['git', 'pull', gistUrl])
			os.chdir(startd)
		else:
			call(['git', 'clone', gistUrl])
		if gist['description'] == None:
			description = ''
		else:
			description = gist['description'].encode('utf8').replace("\r",' ').replace("\n",' ')
		print >> f, gist['id'], gistUrl, description

backup private and public gists

在備份個人gist資料時，當然是希望pirvate和public都可以一起備份。這個就比較複雜點，必需要使用v3的API走過OAuth來取得授權TOKEN。

https://github.com/aprescott/gist-backup/blob/master/gist-backup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
#!/usr/bin/env bash
#
# gist-backup
#
# Backs up all your public and private gists
# to the given directory.
#
# Example:
#
#   gist-backup ~/gist-backups
#
# In addition to your own gists you can also backup your starred gists
# or (public) gists of a defined user by putting the gists url in an 
# environment variable.
#
# Example:
#
#   GIST_URL=https://api.github.com/gists/starred gist-backup ~/gist-backups/starred
#   GIST_URL=https://api.github.com/users/aprescott/gists gist-backup ~/gist-backups/aprescott
#

token=$(git config --get github.gist.oauth.token)
url=${GIST_URL:-https://api.github.com/gists}

usage() {
cat <<END_USAGE

Usage:

  gist-backup DIRECTORY

Example:

  gist-backup ~/gist-backups

END_USAGE

if [ -z $token ]
then

cat <<END_USAGE_NOTE
NOTE:

  In order for this to work, you're going to need
  an OAuth token set in your git config under
  
    github.gist.oauth.token
  
  This is because the API token for your account
  cannot be used as authentication/authorization
  in version 3 of GitHub's API.

END_USAGE_NOTE
fi
}

if [ -z $token ]
then
  echo "No OAuth token found in github.gist.oauth.token git config."
  exit 1
fi

# if no backup directory has been given
if [ -z "$1" ]
then
  echo "No backup directory given."
  usage
  exit 1
fi

# if the given directory doesn't exist, create it
if [ ! -e "$1" ]
then
  mkdir -p $1
fi

# go into the given backup directory
cd $1

# if we failed to cd for some reason, abort
if [ $? -gt 0 ]
then
  exit 1
fi

# Take's a git remote URI and clones it into
# the backup directory. If that directory
# exists already, cd's into it and does a
# git pull.
backup() {
  echo "Backing up $1"
  local dir=$(echo "$1" | cut -d / -f 4 | cut -d . -f 1)

  if [ -e $dir ]
  then
    echo "  Already cloned at $PWD/$dir. Pulling."
    cd $dir
    git pull -q
    cd $OLDPWD
  else
    git clone -q $1
  fi
}

page=1
retries=0
MAX_RETRIES=5
while [ $retries -lt $MAX_RETRIES ]
do
  echo "Requesting Gist page: $page from $url"

  gists=$(
  curl -s -H "Authorization: token $token" -d "page=$page" -G $url |
  sed -n 's/.*git_pull_url": "\(.*\)",/\1/p'
  )

  if [ -z "$gists" ]
  then
    echo "No gists found on this page. Trying again."
    retries=$(( retries + 1 ))
    continue
  fi

  for gist in $gists
  do
    backup $gist
  done

  page=$(( page + 1 ))
  retries=0
done

echo "No gists found (anymore). Not trying again."
exit 0

script部份使用aprescott寫的gist-backup，但在之前要先得到Token塞到git設定裡。

1
2
3
4
5
$ curl -u github-user-name:github-password \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"scopes":["gist"], "note": "gist backup"}' \
  https://api.github.com/authorizations

OTP

如果你的github帳號有設定較安全的2FA，這時光靠username和password是不夠的，github會要你再提供2FA的OTP。

1
2
3
4
{
  "message": "Must specify two-factor authentication OTP code.",
  "documentation_url": "https://developer.github.com/v3/auth#working-with-two-factor-authentication"
}

就照文件說明在header裡多塞一個X-GitHub-OTP

1
2
3
4
5
6
$ curl -u github-user-name:github-password \
  -H "Content-Type: application/json" \
  -H "X-GitHub-OTP:012345" \
  -X POST \
  -d '{"scopes":["gist"], "note": "gist backup"}' \
  https://api.github.com/authorizations

一切正常，則返回的一串json裡會有個叫token的欄位，把該欄位的資料複製下

1
$ git config --global github.gist.oauth.token TOKEN

進到 https://github.com/settings/tokens/ 也可以看到有產生一個授權存取gist資料的personal access token。

全部準備完了，就可以直接跑gist-backup sciript把全部資料都爬下來。

1
2
3
4
wget https://raw.githubusercontent.com/aprescott/gist-backup/master/gist-backup
chmod 755 gist-backup
mkdir gist-backup-folder
./gist-backup ./gist-backup-folder