Covid-19 Bereaved Families for Justice, which represents around 3,000 people who have lost a loved one to the virus, said they have repeatedly invited the prime minister to visit the mural at Thames Embankment.
Johnson, who Downing Street confirmed went to the wall late on Tuesday for “quiet reflection”, has been accused of an attempt to “dodge” families angry with his handling of the pandemic.
It came after the PM faced a torrent of criticism when several reports claimed he said last year that bodies could “pile high” before he would order a third national lockdown in England.
Downing Street has also been unable to deny that Johnson said the virus could “let rip” in a separate alleged incident.
Matt Fowler, co-founder of Covid-19 Bereaved Families for Justice, said he had been asking the prime minister to visit the wall “for weeks” and to meet families.
“He’s refused to even acknowledge our request,” he said.
“Then, the day after it’s revealed he said he’d let ‘bodies pile high’ he makes a late evening visit under cover of darkness, just so that he can dodge meeting bereaved families.
“This is a cynical and insincere move that is deeply hurtful. Our invitation for him to walk the wall with families who’ve lost loved ones is still open, and we await a response.”
Labour MP Chris Matheson told HuffPost UK: “I saw him there as I walked home.
“There was no press, just his security officers. I nodded and said hello and went on my way.”
A No 10 spokeswoman confirmed Johnson was there, telling reporters: “The prime minister offers his deepest condolences to everyone who has lost a loved one during this very difficult pandemic.
“On Tuesday, the prime minister visited the Covid Memorial Wall in private for quiet reflection.”
Journalists have cited a number of different sources saying Johnson made the “pile high” comment.
But the PM has denied it both in interviews and in the Commons on Wednesday.
Labour leader Keir Starmer said any minister who lied in parliament would be “expected to offer their resignation”. He told Johnson during PMQs: “Somebody here isn’t telling the truth.”
The wall, on the south side of the Thames opposite the Houses of Parliament, features roughly 150,000 pink and red hearts representing those who have died with coronavirus.
It stretches almost 500 metres between Westminster and Lambeth bridges and is still having news hearts added.
The volunteers who created it have argued the memorial should become a permenant fixture of the city.
A number of prominent public figures, including London mayor Sadiq Khan and archbishop of Canterbury Justin Welby, have visited the mural in recent days.